Oct 30 22:06:13.312538 ip-10-0-142-185 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 22:06:13.704939 ip-10-0-142-185 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:06:13.704939 ip-10-0-142-185 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 22:06:13.704939 ip-10-0-142-185 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:06:13.704939 ip-10-0-142-185 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:06:13.709382 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709297    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:13.709382 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709381    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:13.709382 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709385    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:13.709471 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709389    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:13.709471 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709392    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:13.709471 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709395    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:13.709471 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709397    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:13.709471 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709400    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:13.709471 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709403    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:13.709471 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709405    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:13.709471 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709408    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:13.709471 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709410    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:13.709471 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709413    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:13.709471 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709415    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:13.709471 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709418    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:13.709471 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709420    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:13.709471 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709423    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:13.709471 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709425    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:13.709471 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709428    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:13.709471 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709430    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:13.709471 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709433    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:13.709471 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709436    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:13.709471 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709439    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:13.709924 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709441    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:13.709924 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709444    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:13.709924 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709446    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:13.709924 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709452    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:13.709924 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709455    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:13.709924 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709458    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:13.709924 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709460    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:13.709924 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709463    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:13.709924 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709465    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:13.709924 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709468    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:13.709924 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709470    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:13.709924 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709472    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:13.709924 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709475    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:13.709924 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709478    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:13.709924 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709481    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:13.709924 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709484    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:13.709924 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709487    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:13.709924 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709489    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:13.709924 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709492    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:13.710380 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709494    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:13.710380 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709496    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:13.710380 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709499    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:13.710380 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709502    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:13.710380 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709504    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:13.710380 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709507    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:13.710380 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709509    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:13.710380 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709512    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:13.710380 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709515    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:13.710380 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709518    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:13.710380 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709520    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:13.710380 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709523    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:13.710380 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709525    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:13.710380 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709528    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:13.710380 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709531    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:13.710380 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709534    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:13.710380 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709536    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:13.710380 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709539    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:13.710380 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709541    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:13.710930 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709544    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:13.710930 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709546    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:13.710930 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709549    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:13.710930 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709551    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:13.710930 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709553    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:13.710930 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709556    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:13.710930 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709558    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:13.710930 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709560    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:13.710930 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709563    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:13.710930 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709565    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:13.710930 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709568    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:13.710930 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709570    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:13.710930 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709573    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:13.710930 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709575    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:13.710930 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709578    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:13.710930 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709581    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:13.710930 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709583    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:13.710930 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709586    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:13.710930 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709588    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:13.710930 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709590    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:13.711419 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709595    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:13.711419 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709599    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:13.711419 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709602    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:13.711419 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709606    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:13.711419 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709609    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:13.711419 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709612    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:13.711419 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709615    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:13.711419 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709620    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:13.711419 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709623    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:13.711419 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709626    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:13.711419 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709628    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:13.711419 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709631    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:13.711419 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709634    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:13.711419 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709636    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:13.711419 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709639    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:13.711419 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709641    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:13.711419 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709643    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:13.711419 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709647    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:13.711419 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709652    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:13.711870 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709656    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:13.711870 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709659    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:13.711870 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709661    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:13.711870 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709664    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:13.711870 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709666    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:13.711870 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.709669    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:13.711870 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710463    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:13.711870 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710469    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:13.711870 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710472    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:13.711870 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710475    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:13.711870 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710477    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:13.711870 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710480    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:13.711870 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710483    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:13.711870 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710485    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:13.711870 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710488    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:13.711870 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710490    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:13.711870 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710493    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:13.711870 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710496    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:13.711870 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710499    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:13.711870 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710504    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:13.712356 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710508    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:13.712356 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710511    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:13.712356 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710513    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:13.712356 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710516    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:13.712356 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710518    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:13.712356 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710521    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:13.712356 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710524    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:13.712356 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710526    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:13.712356 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710528    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:13.712356 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710533    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:13.712356 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710536    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:13.712356 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710539    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:13.712356 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710543    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:13.712356 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710546    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:13.712356 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710549    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:13.712356 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710551    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:13.712356 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710554    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:13.712356 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710557    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:13.712356 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710559    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:13.712802 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710562    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:13.712802 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710564    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:13.712802 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710567    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:13.712802 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710569    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:13.712802 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710572    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:13.712802 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710574    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:13.712802 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710577    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:13.712802 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710579    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:13.712802 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710582    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:13.712802 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710584    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:13.712802 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710587    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:13.712802 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710590    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:13.712802 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710595    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:13.712802 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710598    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:13.712802 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710600    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:13.712802 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710603    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:13.712802 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710606    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:13.712802 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710608    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:13.712802 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710610    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:13.713256 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710613    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:13.713256 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710616    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:13.713256 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710618    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:13.713256 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710621    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:13.713256 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710623    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:13.713256 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710625    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:13.713256 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710628    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:13.713256 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710631    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:13.713256 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710633    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:13.713256 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710635    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:13.713256 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710639    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:13.713256 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710641    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:13.713256 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710644    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:13.713256 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710646    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:13.713256 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710649    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:13.713256 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710651    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:13.713256 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710654    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:13.713256 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710656    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:13.713256 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710659    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:13.713694 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710662    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:13.713694 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710664    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:13.713694 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710666    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:13.713694 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710669    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:13.713694 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710671    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:13.713694 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710673    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:13.713694 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710677    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:13.713694 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710680    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:13.713694 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710683    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:13.713694 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710686    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:13.713694 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710689    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:13.713694 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710692    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:13.713694 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710695    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:13.713694 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710698    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:13.713694 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710700    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:13.713694 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710703    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:13.713694 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710705    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:13.713694 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710708    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:13.713694 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710710    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:13.713694 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710712    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:13.714157 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710715    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:13.714157 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710717    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:13.714157 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710720    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:13.714157 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710723    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:13.714157 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710726    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:13.714157 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710728    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:13.714157 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710731    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:13.714157 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710733    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:13.714157 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710736    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:13.714157 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710738    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:13.714157 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710741    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:13.714157 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710744    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:13.714157 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710747    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:13.714157 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710749    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:13.714157 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.710751    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:13.714157 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711253    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:13.714157 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711259    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:13.714157 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711261    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:13.714157 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711266    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:13.714633 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711269    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:13.714633 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711272    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:13.714633 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711275    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:13.714633 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711277    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:13.714633 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711280    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:13.714633 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711283    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:13.714633 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711285    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:13.714633 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711288    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:13.714633 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711291    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:13.714633 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711293    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:13.714633 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711296    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:13.714633 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711298    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:13.714633 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711301    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:13.714633 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711303    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:13.714633 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711306    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:13.714633 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711308    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:13.714633 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711311    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:13.714633 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711313    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:13.714633 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711316    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:13.714633 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711318    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:13.715097 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711320    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:13.715097 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711323    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:13.715097 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711326    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:13.715097 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711328    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:13.715097 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711331    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:13.715097 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711333    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:13.715097 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711336    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:13.715097 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711338    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:13.715097 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711340    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:13.715097 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711343    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:13.715097 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711345    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:13.715097 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711351    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:13.715097 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711355    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:13.715097 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711359    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:13.715097 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711362    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:13.715097 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711365    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:13.715097 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711368    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:13.715097 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711371    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:13.715097 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711374    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:13.715976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711377    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:13.715976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711379    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:13.715976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711382    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:13.715976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711384    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:13.715976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711387    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:13.715976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711390    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:13.715976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711392    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:13.715976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711395    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:13.715976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711397    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:13.715976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711400    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:13.715976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711402    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:13.715976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711405    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:13.715976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711407    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:13.715976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711410    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:13.715976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711412    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:13.715976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711414    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:13.715976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711417    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:13.715976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711419    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:13.715976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711422    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:13.716452 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711425    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:13.716452 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711427    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:13.716452 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711430    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:13.716452 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711433    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:13.716452 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711435    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:13.716452 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711440    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:13.716452 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711442    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:13.716452 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711445    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:13.716452 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711448    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:13.716452 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711450    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:13.716452 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711453    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:13.716452 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711455    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:13.716452 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711458    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:13.716452 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711460    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:13.716452 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711462    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:13.716452 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711465    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:13.716452 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711467    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:13.716452 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711470    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:13.716452 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711472    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:13.716452 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711474    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:13.716944 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711478    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:13.716944 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711481    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:13.716944 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711485    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:13.716944 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711488    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:13.716944 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711492    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:13.716944 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711494    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:13.716944 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711497    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:13.716944 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711499    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:13.716944 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711502    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:13.716944 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711504    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:13.716944 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711507    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:13.716944 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711509    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:13.716944 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711512    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:13.716944 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711515    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:13.716944 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711517    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:13.716944 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711519    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:13.716944 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711522    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:13.716944 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711526    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:13.716944 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711529    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:13.717401 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711531    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:13.717401 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711534    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:13.717401 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711537    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:13.717401 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711539    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:13.717401 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.711542    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:13.717401 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712336    2580 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 22:06:13.717401 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712347    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 22:06:13.717401 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712353    2580 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 22:06:13.717401 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712358    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 22:06:13.717401 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712363    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 22:06:13.717401 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712366    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 22:06:13.717401 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712371    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 22:06:13.717401 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712376    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 22:06:13.717401 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712379    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 22:06:13.717401 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712382    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 22:06:13.717401 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712386    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 22:06:13.717401 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712390    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 22:06:13.717401 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712393    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 22:06:13.717401 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712395    2580 flags.go:64] FLAG: --cgroup-root=""
Oct 30 22:06:13.717401 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712398    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 22:06:13.717401 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712402    2580 flags.go:64] FLAG: --client-ca-file=""
Oct 30 22:06:13.717401 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712404    2580 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 22:06:13.717401 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712407    2580 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 22:06:13.717929 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712413    2580 flags.go:64] FLAG: --cluster-domain=""
Oct 30 22:06:13.717929 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712415    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 22:06:13.717929 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712419    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 22:06:13.717929 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712422    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 22:06:13.717929 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712426    2580 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 22:06:13.717929 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712431    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 22:06:13.717929 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712434    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 22:06:13.717929 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712439    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 22:06:13.717929 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712442    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 22:06:13.717929 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712446    2580 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 22:06:13.717929 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712449    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 22:06:13.717929 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712453    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 22:06:13.717929 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712456    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 22:06:13.717929 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712460    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 22:06:13.717929 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712464    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 22:06:13.717929 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712467    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 22:06:13.717929 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712470    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 22:06:13.717929 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712473    2580 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 22:06:13.717929 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712476    2580 flags.go:64] FLAG: --enable-server="true"
Oct 30 22:06:13.717929 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712479    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 22:06:13.717929 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712483    2580 flags.go:64] FLAG: --event-burst="100"
Oct 30 22:06:13.717929 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712486    2580 flags.go:64] FLAG: --event-qps="50"
Oct 30 22:06:13.717929 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712489    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 22:06:13.717929 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712492    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 22:06:13.718513 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712494    2580 flags.go:64] FLAG: --eviction-hard=""
Oct 30 22:06:13.718513 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712499    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 22:06:13.718513 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712502    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 22:06:13.718513 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712505    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 22:06:13.718513 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712508    2580 flags.go:64] FLAG: --eviction-soft=""
Oct 30 22:06:13.718513 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712511    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 22:06:13.718513 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712513    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 22:06:13.718513 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712516    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 22:06:13.718513 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712519    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 22:06:13.718513 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712522    2580 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 22:06:13.718513 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712525    2580 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 22:06:13.718513 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712528    2580 flags.go:64] FLAG: --feature-gates=""
Oct 30 22:06:13.718513 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712532    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 22:06:13.718513 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712535    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 22:06:13.718513 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712538    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 22:06:13.718513 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712542    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 22:06:13.718513 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712545    2580 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 22:06:13.718513 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712548    2580 flags.go:64] FLAG: --help="false"
Oct 30 22:06:13.718513 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712551    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-142-185.ec2.internal"
Oct 30 22:06:13.718513 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712554    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 22:06:13.718513 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712557    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 22:06:13.718513 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712560    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 22:06:13.718513 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712564    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 22:06:13.718513 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712567    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 22:06:13.719069 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712570    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 22:06:13.719069 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712573    2580 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 22:06:13.719069 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712575    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 22:06:13.719069 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712578    2580 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 22:06:13.719069 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712581    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 22:06:13.719069 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712584    2580 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 22:06:13.719069 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712587    2580 flags.go:64] FLAG: --kube-reserved=""
Oct 30 22:06:13.719069 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712590    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 22:06:13.719069 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712592    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 22:06:13.719069 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712595    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 22:06:13.719069 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712598    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 22:06:13.719069 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712601    2580 flags.go:64] FLAG: --lock-file=""
Oct 30 22:06:13.719069 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712603    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 22:06:13.719069 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712606    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 22:06:13.719069 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712609    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 22:06:13.719069 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712615    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 22:06:13.719069 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712617    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 22:06:13.719069 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712622    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 22:06:13.719069 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712625    2580 flags.go:64] FLAG: --logging-format="text"
Oct 30 22:06:13.719069 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712628    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 22:06:13.719069 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712631    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 22:06:13.719069 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712634    2580 flags.go:64] FLAG: --manifest-url=""
Oct 30 22:06:13.719069 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712637    2580 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 22:06:13.719069 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712642    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 22:06:13.719069 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712645    2580 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 22:06:13.719687 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712649    2580 flags.go:64] FLAG: --max-pods="110"
Oct 30 22:06:13.719687 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712652    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 22:06:13.719687 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712655    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 22:06:13.719687 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712658    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 22:06:13.719687 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712662    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 22:06:13.719687 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712665    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 22:06:13.719687 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712668    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 22:06:13.719687 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712670    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 22:06:13.719687 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712677    2580 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 22:06:13.719687 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712681    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 22:06:13.719687 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712684    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 22:06:13.719687 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712687    2580 flags.go:64] FLAG: --pod-cidr=""
Oct 30 22:06:13.719687 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712689    2580 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 22:06:13.719687 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712692    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 22:06:13.719687 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712695    2580 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 22:06:13.719687 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712698    2580 flags.go:64] FLAG: --port="10250"
Oct 30 22:06:13.719687 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712701    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 22:06:13.719687 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712704    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0585283188777a4e7"
Oct 30 22:06:13.719687 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712707    2580 flags.go:64] FLAG: --qos-reserved=""
Oct 30 22:06:13.719687 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712710    2580 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 22:06:13.719687 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712713    2580 flags.go:64] FLAG: --register-node="true"
Oct 30 22:06:13.719687 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712716    2580 flags.go:64] FLAG: --register-with-taints=""
Oct 30 22:06:13.719687 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712720    2580 flags.go:64] FLAG: --registry-burst="10"
Oct 30 22:06:13.719687 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712723    2580 flags.go:64] FLAG: --registry-qps="5"
Oct 30 22:06:13.719687 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712727    2580 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 22:06:13.719687 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712730    2580 flags.go:64] FLAG: --reserved-memory=""
Oct 30 22:06:13.721385 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712734    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 22:06:13.721385 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712737    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 22:06:13.721385 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712740    2580 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 22:06:13.721385 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712742    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 22:06:13.721385 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712747    2580 flags.go:64] FLAG: --runonce="false"
Oct 30 22:06:13.721385 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712750    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 22:06:13.721385 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712753    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 22:06:13.721385 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712756    2580 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 22:06:13.721385 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712759    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 22:06:13.721385 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712762    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 22:06:13.721385 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712765    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 22:06:13.721385 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712768    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 22:06:13.721385 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712771    2580 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 22:06:13.721385 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712774    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 22:06:13.721385 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712777    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 22:06:13.721385 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712780    2580 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 22:06:13.721385 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712783    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 22:06:13.721385 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712786    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 22:06:13.721385 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712789    2580 flags.go:64] FLAG: --system-cgroups=""
Oct 30 22:06:13.721385 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712792    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 22:06:13.721385 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712798    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 22:06:13.721385 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712801    2580 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 22:06:13.721385 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712804    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 22:06:13.721385 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712808    2580 flags.go:64] FLAG: --tls-min-version=""
Oct 30 22:06:13.721385 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712811    2580 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 22:06:13.722177 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712814    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 22:06:13.722177 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712817    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 22:06:13.722177 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712820    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 22:06:13.722177 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712823    2580 flags.go:64] FLAG: --v="2"
Oct 30 22:06:13.722177 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712827    2580 flags.go:64] FLAG: --version="false"
Oct 30 22:06:13.722177 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712833    2580 flags.go:64] FLAG: --vmodule=""
Oct 30 22:06:13.722177 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712837    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 22:06:13.722177 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.712841    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 22:06:13.722177 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713010    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:13.722177 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713015    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:13.722177 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713018    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:13.722177 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713021    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:13.722177 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713024    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:13.722177 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713027    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:13.722177 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713029    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:13.722177 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713032    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:13.722177 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713036    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:13.722177 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713039    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:13.722177 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713042    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:13.722177 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713046    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:13.722177 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713048    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:13.722177 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713051    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:13.722741 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713053    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:13.722741 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713056    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:13.722741 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713059    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:13.722741 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713061    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:13.722741 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713063    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:13.722741 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713066    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:13.722741 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713068    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:13.722741 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713071    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:13.722741 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713074    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:13.722741 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713076    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:13.722741 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713079    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:13.722741 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713081    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:13.722741 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713083    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:13.722741 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713086    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:13.722741 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713089    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:13.722741 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713092    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:13.722741 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713095    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:13.722741 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713097    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:13.722741 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713100    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:13.722741 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713103    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:13.723320 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713105    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:13.723320 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713108    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:13.723320 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713110    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:13.723320 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713113    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:13.723320 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713115    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:13.723320 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713118    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:13.723320 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713120    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:13.723320 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713123    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:13.723320 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713125    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:13.723320 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713128    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:13.723320 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713130    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:13.723320 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713133    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:13.723320 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713135    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:13.723320 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713138    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:13.723320 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713140    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:13.723320 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713144    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:13.723320 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713148    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:13.723320 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713151    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:13.723320 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713154    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:13.723771 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713157    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:13.723771 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713160    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:13.723771 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713162    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:13.723771 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713165    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:13.723771 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713167    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:13.723771 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713170    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:13.723771 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713172    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:13.723771 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713175    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:13.723771 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713178    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:13.723771 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713181    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:13.723771 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713183    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:13.723771 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713186    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:13.723771 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713189    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:13.723771 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713191    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:13.723771 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713194    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:13.723771 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713196    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:13.723771 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713199    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:13.723771 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713201    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:13.723771 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713203    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:13.723771 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713206    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:13.724264 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713209    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:13.724264 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713211    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:13.724264 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713213    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:13.724264 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713216    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:13.724264 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713231    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:13.724264 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713234    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:13.724264 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713236    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:13.724264 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713239    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:13.724264 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713242    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:13.724264 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713244    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:13.724264 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713247    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:13.724264 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713249    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:13.724264 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713251    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:13.724264 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713254    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:13.724264 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713256    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:13.724264 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713259    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:13.724264 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713261    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:13.724264 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713264    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:13.724264 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713266    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:13.724700 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713269    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:13.724700 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713273    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:13.724700 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713275    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:13.724700 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713278    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:13.724700 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713281    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:13.724700 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713283    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:13.724700 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713286    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:13.724700 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713289    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:13.724700 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713291    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:13.724700 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713294    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:13.724700 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713296    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:13.724700 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713299    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:13.724700 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713301    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:13.724700 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713304    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:13.724700 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.713331    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:06:13.724700 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713366    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:13.725085 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713370    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:13.725085 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713373    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:13.725085 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713376    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:13.725085 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713379    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:13.725085 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713382    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:13.725085 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713385    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:13.725085 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713388    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:13.725085 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713391    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:13.725085 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713393    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:13.725085 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713396    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:13.725085 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713398    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:13.725085 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713401    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:13.725085 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713403    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:13.725085 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713406    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:13.725085 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713408    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:13.725085 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713411    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:13.725085 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713413    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:13.725085 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713416    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:13.725085 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713419    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:13.725540 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713422    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:13.725540 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713424    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:13.725540 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713426    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:13.725540 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713429    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:13.725540 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713433    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:13.725540 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713435    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:13.725540 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713438    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:13.725540 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713440    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:13.725540 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713443    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:13.725540 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713445    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:13.725540 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713448    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:13.725540 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713451    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:13.725540 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713453    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:13.725540 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713456    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:13.725540 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713458    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:13.725540 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713461    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:13.725540 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713464    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:13.725540 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713466    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:13.725540 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713469    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:13.725976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713471    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:13.725976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713474    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:13.725976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713476    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:13.725976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713479    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:13.725976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713481    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:13.725976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713483    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:13.725976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713486    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:13.725976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713488    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:13.725976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713490    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:13.725976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713494    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:13.725976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713498    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:13.725976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713501    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:13.725976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713505    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:13.725976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713507    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:13.725976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713510    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:13.725976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713513    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:13.725976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713516    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:13.725976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713520    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:13.725976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713522    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:13.725976 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713525    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:13.726454 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713528    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:13.726454 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713531    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:13.726454 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713534    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:13.726454 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713536    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:13.726454 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713539    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:13.726454 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713541    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:13.726454 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713544    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:13.726454 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713546    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:13.726454 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713549    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:13.726454 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713551    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:13.726454 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713555    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:13.726454 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713557    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:13.726454 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713560    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:13.726454 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713562    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:13.726454 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713564    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:13.726454 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713567    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:13.726454 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713569    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:13.726454 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713572    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:13.726454 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713574    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:13.726454 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713577    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:13.726927 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713579    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:13.726927 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713582    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:13.726927 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713584    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:13.726927 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713587    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:13.726927 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713591    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:13.726927 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713594    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:13.726927 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713596    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:13.726927 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713599    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:13.726927 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713601    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:13.726927 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713605    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:13.726927 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713609    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:13.726927 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713612    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:13.726927 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713615    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:13.726927 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713618    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:13.726927 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713620    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:13.726927 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713623    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:13.726927 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713625    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:13.726927 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713628    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:13.726927 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713630    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:13.727409 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713633    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:13.727409 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713635    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:13.727409 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713638    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:13.727409 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713640    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:13.727409 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713644    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:13.727409 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713647    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:13.727409 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713650    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:13.727409 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.713652    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:13.727409 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.721343    2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 22:06:13.727409 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.721382    2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 22:06:13.727409 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.721405    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 22:06:13.727409 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.721414    2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 22:06:13.727409 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721583    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:13.727409 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721591    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:13.727409 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721597    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:13.727409 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721601    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:13.727409 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721606    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:13.727409 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721615    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:13.727409 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721619    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:13.727409 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721624    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:13.727882 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721629    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:13.727882 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721634    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:13.727882 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721638    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:13.727882 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721643    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:13.727882 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721647    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:13.727882 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721651    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:13.727882 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721655    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:13.727882 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721659    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:13.727882 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721664    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:13.727882 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721673    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:13.727882 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721677    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:13.727882 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721682    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:13.727882 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721686    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:13.727882 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721691    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:13.727882 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721695    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:13.727882 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721700    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:13.727882 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721704    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:13.727882 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721708    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:13.727882 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721712    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:13.728376 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721716    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:13.728376 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721720    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:13.728376 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721726    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:13.728376 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721735    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:13.728376 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721740    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:13.728376 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721744    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:13.728376 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721750    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:13.728376 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721757    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:13.728376 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721762    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:13.728376 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721768    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:13.728376 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721773    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:13.728376 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721778    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:13.728376 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721782    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:13.728376 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721786    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:13.728376 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721790    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:13.728376 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721794    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:13.728376 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721803    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:13.728376 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721807    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:13.728376 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721811    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:13.728809 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721815    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:13.728809 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721819    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:13.728809 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721823    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:13.728809 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721828    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:13.728809 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721832    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:13.728809 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721836    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:13.728809 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721875    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:13.728809 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721882    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:13.728809 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721887    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:13.728809 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721893    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:13.728809 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721899    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:13.728809 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721904    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:13.728809 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721909    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:13.728809 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721913    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:13.728809 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721918    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:13.728809 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721931    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:13.728809 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.721934    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:13.728809 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722051    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:13.728809 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722056    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:13.728809 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722059    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:13.729297 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722062    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:13.729297 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722066    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:13.729297 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722069    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:13.729297 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722073    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:13.729297 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722076    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:13.729297 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722078    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:13.729297 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722081    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:13.729297 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722084    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:13.729297 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722087    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:13.729297 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722090    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:13.729297 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722093    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:13.729297 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722096    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:13.729297 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722098    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:13.729297 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722101    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:13.729297 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722103    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:13.729297 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722106    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:13.729297 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722108    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:13.729297 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722111    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:13.729297 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722114    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:13.729743 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722116    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:13.729743 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722119    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:13.729743 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722121    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:13.729743 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722124    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:13.729743 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722126    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:13.729743 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722129    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:13.729743 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722131    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:13.729743 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722134    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:13.729743 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722136    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:13.729743 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722139    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:13.729743 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722141    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:13.729743 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722144    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:13.729743 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722146    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:13.729743 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722149    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:13.729743 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722152    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:13.729743 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722155    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:13.729743 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722157    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:13.729743 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722160    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:13.729743 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722162    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:13.729743 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722165    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:13.730198 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722167    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:13.730198 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.722203    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:06:13.730198 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722356    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:13.730198 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722361    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:13.730198 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722364    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:13.730198 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722367    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:13.730198 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722369    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:13.730198 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722372    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:13.730198 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722374    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:13.730198 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722377    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:13.730198 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722379    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:13.730198 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722382    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:13.730198 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722385    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:13.730198 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722387    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:13.730198 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722390    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:13.730198 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722393    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:13.730597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722395    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:13.730597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722398    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:13.730597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722401    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:13.730597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722403    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:13.730597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722406    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:13.730597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722408    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:13.730597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722411    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:13.730597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722413    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:13.730597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722416    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:13.730597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722418    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:13.730597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722420    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:13.730597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722423    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:13.730597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722425    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:13.730597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722427    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:13.730597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722430    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:13.730597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722433    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:13.730597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722437    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:13.730597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722440    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:13.730597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722443    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:13.730597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722446    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:13.731058 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722448    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:13.731058 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722451    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:13.731058 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722453    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:13.731058 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722455    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:13.731058 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722458    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:13.731058 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722460    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:13.731058 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722462    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:13.731058 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722465    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:13.731058 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722468    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:13.731058 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722470    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:13.731058 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722472    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:13.731058 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722475    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:13.731058 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722478    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:13.731058 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722480    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:13.731058 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722483    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:13.731058 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722485    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:13.731058 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722487    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:13.731058 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722490    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:13.731058 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722492    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:13.731058 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722495    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:13.731623 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722497    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:13.731623 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722499    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:13.731623 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722502    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:13.731623 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722504    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:13.731623 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722506    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:13.731623 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722509    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:13.731623 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722511    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:13.731623 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722514    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:13.731623 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722516    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:13.731623 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722518    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:13.731623 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722521    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:13.731623 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722523    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:13.731623 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722526    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:13.731623 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722528    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:13.731623 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722530    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:13.731623 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722533    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:13.731623 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722537    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:13.731623 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722541    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:13.731623 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722545    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:13.732103 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722548    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:13.732103 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722551    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:13.732103 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722553    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:13.732103 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722556    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:13.732103 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722558    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:13.732103 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722560    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:13.732103 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722563    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:13.732103 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722565    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:13.732103 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722568    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:13.732103 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722570    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:13.732103 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722573    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:13.732103 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722575    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:13.732103 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722578    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:13.732103 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722580    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:13.732103 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722582    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:13.732103 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722585    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:13.732103 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722587    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:13.732103 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722590    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:13.732103 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722592    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:13.732103 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722595    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:13.732597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722597    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:13.732597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722599    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:13.732597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722601    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:13.732597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722604    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:13.732597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722606    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:13.732597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722609    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:13.732597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722611    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:13.732597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722614    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:13.732597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722616    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:13.732597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722619    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:13.732597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722621    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:13.732597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722624    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:13.732597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722628    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:13.732597 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.722651    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:06:13.732597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722682    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:13.732597 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722686    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:13.732960 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722688    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:13.732960 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722691    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:13.732960 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722694    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:13.732960 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722697    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:13.732960 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722700    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:13.732960 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722702    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:13.732960 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722705    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:13.732960 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722707    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:13.732960 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722709    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:13.732960 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722713    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:13.732960 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722715    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:13.732960 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722718    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:13.732960 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722721    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:13.732960 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722724    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:13.732960 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722726    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:13.732960 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722728    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:13.732960 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722731    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:13.732960 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722733    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:13.732960 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722736    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:13.732960 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722740    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:13.733433 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722744    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:13.733433 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722748    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:13.733433 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722750    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:13.733433 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722753    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:13.733433 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722756    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:13.733433 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722758    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:13.733433 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722760    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:13.733433 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722763    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:13.733433 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722766    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:13.733433 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722769    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:13.733433 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722771    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:13.733433 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722774    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:13.733433 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722776    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:13.733433 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722779    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:13.733433 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722781    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:13.733433 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722783    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:13.733433 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722786    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:13.733433 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722788    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:13.733433 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722791    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:13.733433 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722793    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:13.733894 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722795    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:13.733894 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722798    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:13.733894 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722800    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:13.733894 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722803    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:13.733894 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722805    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:13.733894 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722807    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:13.733894 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722810    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:13.733894 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722812    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:13.733894 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722815    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:13.733894 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722817    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:13.733894 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722820    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:13.733894 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722822    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:13.733894 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722824    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:13.733894 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722827    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:13.733894 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722829    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:13.733894 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722831    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:13.733894 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722834    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:13.733894 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722836    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:13.733894 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722838    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:13.734360 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722843    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:13.734360 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722847    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:13.734360 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722852    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:13.734360 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722855    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:13.734360 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722858    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:13.734360 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722861    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:13.734360 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722864    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:13.734360 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722867    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:13.734360 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722869    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:13.734360 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722872    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:13.734360 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722875    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:13.734360 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722877    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:13.734360 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722880    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:13.734360 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722882    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:13.734360 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722885    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:13.734360 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722887    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:13.734360 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722889    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:13.734360 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722892    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:13.734360 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722894    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:13.734797 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722896    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:13.734797 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722899    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:13.734797 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722902    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:13.734797 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722904    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:13.734797 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722906    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:13.734797 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722909    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:13.734797 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722911    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:13.734797 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722913    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:13.734797 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722916    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:13.734797 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722918    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:13.734797 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722920    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:13.734797 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722923    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:13.734797 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722925    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:13.734797 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722930    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:13.734797 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722933    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:13.734797 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722935    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:13.734797 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722938    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:13.734797 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722941    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:13.734797 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722944    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:13.734797 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722946    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:13.735324 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722948    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:13.735324 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722951    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:13.735324 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722953    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:13.735324 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722956    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:13.735324 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722958    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:13.735324 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:13.722961    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:13.735324 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.723021    2580 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 22:06:13.735324 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.727688    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 22:06:13.735324 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.728584    2580 server.go:1014] "Starting client certificate rotation"
Oct 30 22:06:13.735324 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.728680    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 22:06:13.735324 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.728717    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 22:06:13.735324 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.730316    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 22:06:13.735324 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.731523    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 22:06:13.741136 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.741121    2580 log.go:25] "Validated CRI v1 runtime API"
Oct 30 22:06:13.744789 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.744773    2580 log.go:25] "Validated CRI v1 image API"
Oct 30 22:06:13.745629 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.745598    2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 22:06:13.749638 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.749616    2580 fs.go:135] Filesystem UUIDs: map[2a35b543-9686-42b6-bb7d-21827bdd91c2:/dev/nvme0n1p3 615c480a-1685-45d6-885f-06ce32897ea8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 22:06:13.749713 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.749637    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 22:06:13.755331 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.755192    2580 manager.go:217] Machine: {Timestamp:2026-10-30 22:06:13.753913998 +0000 UTC m=+0.324007284 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099241 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29286363831a8798a39233637872d6 SystemUUID:ec292863-6383-1a87-98a3-9233637872d6 BootID:9fab111f-a068-424a-ac61-d6661241e36c Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ea:b9:24:63 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ea:b9:24:63 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:e8:06:a5:ee:66 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 22:06:13.755331 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.755323    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 22:06:13.755455 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.755432    2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 22:06:13.756383 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.756357    2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 22:06:13.756510 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.756384    2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-185.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 22:06:13.756549 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.756512    2580 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 22:06:13.756549 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.756520    2580 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 22:06:13.756549 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.756545    2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 22:06:13.756629 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.756562    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 22:06:13.757264 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.757255    2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 22:06:13.761207 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.761196    2580 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 22:06:13.761342 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.761333    2580 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 22:06:13.761964 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.761954    2580 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 22:06:13.761991 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.761971    2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 22:06:13.761991 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.761989    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 22:06:13.762044 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.762001    2580 kubelet.go:405] "Adding apiserver pod source"
Oct 30 22:06:13.762044 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.762012    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 22:06:13.762418 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.762404    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 22:06:13.765165 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.765139    2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 22:06:13.776883 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.776867    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 22:06:13.777746 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.777733    2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 22:06:13.777790 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.777765    2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 22:06:13.777934 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.777924    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 22:06:13.777962 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.777941    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 22:06:13.777962 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.777949    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 22:06:13.777962 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.777956    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 22:06:13.777962 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.777963    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 22:06:13.778058 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.777970    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 22:06:13.778417 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.778397    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 22:06:13.778417 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.778411    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 22:06:13.778417 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.778419    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 22:06:13.778561 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.778426    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 22:06:13.778561 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.778437    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 22:06:13.778561 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.778448    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 22:06:13.779267 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.779255    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 22:06:13.779307 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.779274    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 22:06:13.783003 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.782990    2580 server.go:1263] "Started kubelet"
Oct 30 22:06:13.783124 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.783075    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 22:06:13.783176 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.783155    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 22:06:13.783218 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.783086    2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 22:06:13.783840 ip-10-0-142-185 systemd[1]: Started Kubernetes Kubelet.
Oct 30 22:06:13.784359 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.784334    2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 22:06:13.785422 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.785409    2580 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 22:06:13.788514 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:13.788493    2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 22:06:13.789158 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.789136    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 22:06:13.789636 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.789616    2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 22:06:13.790039 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:13.790018    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-185.ec2.internal\" not found"
Oct 30 22:06:13.790215 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.790193    2580 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 22:06:13.790215 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.790214    2580 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 22:06:13.790373 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.790196    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 22:06:13.790373 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.790312    2580 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 22:06:13.790373 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.790320    2580 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 22:06:13.790620 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.790605    2580 factory.go:55] Registering systemd factory
Oct 30 22:06:13.790698 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.790622    2580 factory.go:223] Registration of the systemd container factory successfully
Oct 30 22:06:13.790828 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.790815    2580 factory.go:153] Registering CRI-O factory
Oct 30 22:06:13.790877 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.790830    2580 factory.go:223] Registration of the crio container factory successfully
Oct 30 22:06:13.790906 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.790882    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 22:06:13.790906 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.790897    2580 factory.go:103] Registering Raw factory
Oct 30 22:06:13.790906 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.790907    2580 manager.go:1199] Started watching for new ooms in manager
Oct 30 22:06:13.791904 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.791891    2580 manager.go:319] Starting recovery of all containers
Oct 30 22:06:13.802277 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.802239    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 22:06:13.804091 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.803987    2580 manager.go:324] Recovery completed
Oct 30 22:06:13.808489 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.808463    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-185.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 22:06:13.808489 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:13.808465    2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-185.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 22:06:13.809313 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:13.808471    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-185.ec2.internal.18e36df4903d174b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-185.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-185.ec2.internal,},FirstTimestamp:2026-10-30 22:06:13.782959947 +0000 UTC m=+0.353053233,LastTimestamp:2026-10-30 22:06:13.782959947 +0000 UTC m=+0.353053233,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-185.ec2.internal,}"
Oct 30 22:06:13.809780 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.809767    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:06:13.811531 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.811517    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 22:06:13.811590 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.811546    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 22:06:13.811590 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.811556    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeHasSufficientPID"
Oct 30 22:06:13.812207 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.812193    2580 cpu_manager.go:226] "Starting" policy="none"
Oct 30 22:06:13.812207 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.812204    2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 22:06:13.812362 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.812245    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:06:13.812362 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.812247    2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 22:06:13.812907 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.812894    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 22:06:13.812964 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.812918    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 22:06:13.812964 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.812928    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeHasSufficientPID"
Oct 30 22:06:13.814503 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.814490    2580 policy_none.go:50] "Start"
Oct 30 22:06:13.814552 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.814508    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 22:06:13.814552 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.814518    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 22:06:13.816525 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.816513    2580 policy_none.go:44] "Start"
Oct 30 22:06:13.829317 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.829290    2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 22:06:13.830620 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:13.830592    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-185.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 22:06:13.830738 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:13.830616    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 22:06:13.830804 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:13.830652    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-185.ec2.internal.18e36df491f113c9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-185.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-185.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-185.ec2.internal,},FirstTimestamp:2026-10-30 22:06:13.811532745 +0000 UTC m=+0.381626030,LastTimestamp:2026-10-30 22:06:13.811532745 +0000 UTC m=+0.381626030,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-185.ec2.internal,}"
Oct 30 22:06:13.830804 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:13.830765    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 22:06:13.852948 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:13.850707    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-185.ec2.internal.18e36df491f1583e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-185.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-185.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-185.ec2.internal,},FirstTimestamp:2026-10-30 22:06:13.81155027 +0000 UTC m=+0.381643555,LastTimestamp:2026-10-30 22:06:13.81155027 +0000 UTC m=+0.381643555,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-185.ec2.internal,}"
Oct 30 22:06:13.877479 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.856887    2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 22:06:13.877479 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.859764    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ckfv5"
Oct 30 22:06:13.877479 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.862100    2580 manager.go:341] "Starting Device Plugin manager"
Oct 30 22:06:13.877479 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:13.862126    2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 22:06:13.877479 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.862136    2580 server.go:87] "Starting device plugin registration server"
Oct 30 22:06:13.877479 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.862389    2580 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 22:06:13.877479 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.862400    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 22:06:13.877479 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.862499    2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 22:06:13.877479 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.862578    2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 22:06:13.877479 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.862588    2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 22:06:13.877479 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:13.863182    2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 22:06:13.877479 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:13.863251    2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-185.ec2.internal\" not found"
Oct 30 22:06:13.877479 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.871610    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ckfv5"
Oct 30 22:06:13.877479 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.872298    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 22:06:13.877479 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:13.872462    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-185.ec2.internal.18e36df491f17d13  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-185.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-185.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-185.ec2.internal,},FirstTimestamp:2026-10-30 22:06:13.811559699 +0000 UTC m=+0.381652984,LastTimestamp:2026-10-30 22:06:13.811559699 +0000 UTC m=+0.381652984,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-185.ec2.internal,}"
Oct 30 22:06:13.877479 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.873157    2580 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 22:06:13.877479 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.873182    2580 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 22:06:13.877479 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:13.873215    2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 22:06:13.906498 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.906339    2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:06:13.963388 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.963317    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:06:13.964470 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.964451    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 22:06:13.964576 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.964490    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 22:06:13.964576 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.964509    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeHasSufficientPID"
Oct 30 22:06:13.964576 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.964533    2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-185.ec2.internal"
Oct 30 22:06:13.973735 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.973711    2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-185.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal"]
Oct 30 22:06:13.973784 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.973772    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:06:13.974583 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.974567    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 22:06:13.974671 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.974599    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 22:06:13.974671 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.974615    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeHasSufficientPID"
Oct 30 22:06:13.976941 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.976925    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:06:13.977083 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.977069    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:13.977145 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.977099    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:06:13.977695 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.977679    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 22:06:13.977775 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.977696    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 22:06:13.977775 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.977708    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 22:06:13.977775 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.977715    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 22:06:13.977775 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.977724    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeHasSufficientPID"
Oct 30 22:06:13.977775 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.977728    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeHasSufficientPID"
Oct 30 22:06:13.979883 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.979867    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:13.979966 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.979900    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:06:13.980595 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.980581    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 22:06:13.980681 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.980610    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 22:06:13.980681 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.980624    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeHasSufficientPID"
Oct 30 22:06:13.985881 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:13.985866    2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-185.ec2.internal"
Oct 30 22:06:13.985948 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:13.985887    2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-185.ec2.internal\": node \"ip-10-0-142-185.ec2.internal\" not found"
Oct 30 22:06:14.010013 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:14.009992    2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-185.ec2.internal\" not found" node="ip-10-0-142-185.ec2.internal"
Oct 30 22:06:14.016654 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:14.016636    2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-185.ec2.internal\" not found" node="ip-10-0-142-185.ec2.internal"
Oct 30 22:06:14.032641 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:14.032619    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-185.ec2.internal\" not found"
Oct 30 22:06:14.091526 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:14.091496    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/496cd66cfb3bc1cef71f828d2683a76e-config\") pod \"kube-apiserver-proxy-ip-10-0-142-185.ec2.internal\" (UID: \"496cd66cfb3bc1cef71f828d2683a76e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:14.091684 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:14.091530    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3e2926fda086fe0743710e38cf91711-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal\" (UID: \"c3e2926fda086fe0743710e38cf91711\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:14.091684 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:14.091601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3e2926fda086fe0743710e38cf91711-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal\" (UID: \"c3e2926fda086fe0743710e38cf91711\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:14.133569 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:14.133540    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-185.ec2.internal\" not found"
Oct 30 22:06:14.192303 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:14.192278    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/496cd66cfb3bc1cef71f828d2683a76e-config\") pod \"kube-apiserver-proxy-ip-10-0-142-185.ec2.internal\" (UID: \"496cd66cfb3bc1cef71f828d2683a76e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:14.192401 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:14.192307    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3e2926fda086fe0743710e38cf91711-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal\" (UID: \"c3e2926fda086fe0743710e38cf91711\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:14.192401 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:14.192355    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3e2926fda086fe0743710e38cf91711-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal\" (UID: \"c3e2926fda086fe0743710e38cf91711\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:14.192401 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:14.192384    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/496cd66cfb3bc1cef71f828d2683a76e-config\") pod \"kube-apiserver-proxy-ip-10-0-142-185.ec2.internal\" (UID: \"496cd66cfb3bc1cef71f828d2683a76e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:14.192511 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:14.192389    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3e2926fda086fe0743710e38cf91711-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal\" (UID: \"c3e2926fda086fe0743710e38cf91711\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:14.192511 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:14.192389    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3e2926fda086fe0743710e38cf91711-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal\" (UID: \"c3e2926fda086fe0743710e38cf91711\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:14.234025 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:14.233969    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-185.ec2.internal\" not found"
Oct 30 22:06:14.312826 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:14.312801    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:14.320006 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:14.319992    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:14.335000 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:14.334976    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-185.ec2.internal\" not found"
Oct 30 22:06:14.436012 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:14.435969    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-185.ec2.internal\" not found"
Oct 30 22:06:14.536544 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:14.536495    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-185.ec2.internal\" not found"
Oct 30 22:06:14.637004 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:14.636970    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-185.ec2.internal\" not found"
Oct 30 22:06:14.728499 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:14.728471    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 22:06:14.729005 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:14.728599    2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 22:06:14.737783 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:14.737752    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-185.ec2.internal\" not found"
Oct 30 22:06:14.838595 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:14.838566    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-185.ec2.internal\" not found"
Oct 30 22:06:14.862906 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:14.862876    2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:06:14.876123 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:14.876090    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 22:01:13 +0000 UTC" deadline="2028-04-11 14:56:40.318947242 +0000 UTC"
Oct 30 22:06:14.876123 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:14.876115    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12688h50m25.442834676s"
Oct 30 22:06:14.879913 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:14.879896    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:06:14.880737 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:14.880721    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 22:06:14.880820 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:14.880750    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 22:06:14.880820 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:14.880760    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeHasSufficientPID"
Oct 30 22:06:14.920069 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:14.920043    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e2926fda086fe0743710e38cf91711.slice/crio-940a21805282e57baa73037cb6e8c0073b44aa3db60ee4e376faa4a835331ae7 WatchSource:0}: Error finding container 940a21805282e57baa73037cb6e8c0073b44aa3db60ee4e376faa4a835331ae7: Status 404 returned error can't find the container with id 940a21805282e57baa73037cb6e8c0073b44aa3db60ee4e376faa4a835331ae7
Oct 30 22:06:14.920567 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:14.920547    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496cd66cfb3bc1cef71f828d2683a76e.slice/crio-109c10c088184bcb0a693845404521cea84a57da050db42739d6f87e72638560 WatchSource:0}: Error finding container 109c10c088184bcb0a693845404521cea84a57da050db42739d6f87e72638560: Status 404 returned error can't find the container with id 109c10c088184bcb0a693845404521cea84a57da050db42739d6f87e72638560
Oct 30 22:06:14.926816 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:14.926800    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:06:14.939030 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:14.939014    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-185.ec2.internal\" not found"
Oct 30 22:06:14.992473 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:14.992446    2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:06:15.039847 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:15.039819    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-185.ec2.internal\" not found"
Oct 30 22:06:15.068100 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.068024    2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:06:15.090066 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.090047    2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:15.112888 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.112866    2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:06:15.113748 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.113736    2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:15.138387 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.138371    2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:06:15.592587 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.592515    2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:06:15.763873 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.763848    2580 apiserver.go:52] "Watching apiserver"
Oct 30 22:06:15.790679 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.790655    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 22:06:15.810675 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.810647    2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 22:06:15.811015 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.810993    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-185.ec2.internal","openshift-multus/network-metrics-daemon-fvsmv","openshift-network-operator/iptables-alerter-86vll","kube-system/global-pull-secret-syncer-ljnhp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v","openshift-cluster-node-tuning-operator/tuned-p4g27","openshift-dns/node-resolver-9dqlz","openshift-multus/multus-p8lsb","openshift-multus/multus-additional-cni-plugins-v69rv","openshift-network-diagnostics/network-check-target-zcl2w","openshift-ovn-kubernetes/ovnkube-node-8kfh5","kube-system/konnectivity-agent-2fhd5","openshift-image-registry/node-ca-nwtzc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal"]
Oct 30 22:06:15.813345 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.813325    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9dqlz"
Oct 30 22:06:15.815672 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.815423    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nwtzc"
Oct 30 22:06:15.817595 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.817576    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-86vll"
Oct 30 22:06:15.819686 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.819668    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:15.819784 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:15.819739    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljnhp" podUID="1e444015-5346-4106-93aa-ef43eb9d0be6"
Oct 30 22:06:15.821758 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.821739    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:15.823742 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.823724    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:15.827796 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.825708    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:15.828742 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.828657    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 22:06:15.828825 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.828741    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nn9g8\""
Oct 30 22:06:15.829042 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.829027    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 22:06:15.829129 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.829112    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:15.830864 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.830847    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:15.830974 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:15.830954    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvsmv" podUID="4049e53b-bffe-40fb-beef-22acc171315a"
Oct 30 22:06:15.832783 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.832767    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:15.832864 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:15.832821    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcl2w" podUID="d303600a-2390-4ba9-8ea8-a58bd505a109"
Oct 30 22:06:15.838349 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.837354    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:15.840304 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.839364    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2fhd5"
Oct 30 22:06:15.847402 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.847384    2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 22:06:15.847491 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.847426    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fkrmn"
Oct 30 22:06:15.878350 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.878303    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-185.ec2.internal" event={"ID":"496cd66cfb3bc1cef71f828d2683a76e","Type":"ContainerStarted","Data":"109c10c088184bcb0a693845404521cea84a57da050db42739d6f87e72638560"}
Oct 30 22:06:15.879817 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.879793    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal" event={"ID":"c3e2926fda086fe0743710e38cf91711","Type":"ContainerStarted","Data":"940a21805282e57baa73037cb6e8c0073b44aa3db60ee4e376faa4a835331ae7"}
Oct 30 22:06:15.891552 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.891533    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 22:06:15.901634 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.901609    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:15.901738 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.901646    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swswj\" (UniqueName: \"kubernetes.io/projected/47dc8b20-b520-4f4d-bd48-74b78ba300ef-kube-api-access-swswj\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:15.901738 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.901671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9d1eecff-b22f-4864-a121-9fb507e783f2-konnectivity-ca\") pod \"konnectivity-agent-2fhd5\" (UID: \"9d1eecff-b22f-4864-a121-9fb507e783f2\") " pod="kube-system/konnectivity-agent-2fhd5"
Oct 30 22:06:15.901738 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.901720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10d1cecc-9e30-4354-8b42-60ddceadff44-host\") pod \"node-ca-nwtzc\" (UID: \"10d1cecc-9e30-4354-8b42-60ddceadff44\") " pod="openshift-image-registry/node-ca-nwtzc"
Oct 30 22:06:15.901877 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.901753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-systemd\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:15.901877 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.901810    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-multus-socket-dir-parent\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:15.901877 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.901869    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-cni-bin\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:15.901991 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.901886    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-sys-fs\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:15.901991 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.901911    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfdrz\" (UniqueName: \"kubernetes.io/projected/dd15a503-2539-4458-90cb-dae3e0e28e62-kube-api-access-kfdrz\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:15.901991 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.901928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-run-ovn\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:15.901991 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.901962    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/47dc8b20-b520-4f4d-bd48-74b78ba300ef-ovnkube-script-lib\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:15.902168 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.901997    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/10d1cecc-9e30-4354-8b42-60ddceadff44-serviceca\") pod \"node-ca-nwtzc\" (UID: \"10d1cecc-9e30-4354-8b42-60ddceadff44\") " pod="openshift-image-registry/node-ca-nwtzc"
Oct 30 22:06:15.902168 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.902048    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-var-lib-kubelet\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:15.902168 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.902091    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/47dc8b20-b520-4f4d-bd48-74b78ba300ef-ovn-node-metrics-cert\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:15.902168 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.902114    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-registration-dir\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:15.902168 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.902156    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-device-dir\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:15.902417 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.902193    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-slash\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:15.902417 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.902304    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccf49970-fa36-4188-ab63-e14e440b73ec-tmp\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:15.902417 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.902363    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-etc-kubernetes\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:15.902417 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.902386    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs\") pod \"network-metrics-daemon-fvsmv\" (UID: \"4049e53b-bffe-40fb-beef-22acc171315a\") " pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:15.902582 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.902428    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bdc6308d-a259-45db-90ea-094a8d1dd3a4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:15.902582 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.902451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-run-netns\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:15.902582 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.902517    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-var-lib-cni-bin\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:15.902582 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.902554    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-lib-modules\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:15.902736 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.902590    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-host\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:15.902736 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.902612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2ec19290-25c4-4b52-8173-22c14e7c9056-hosts-file\") pod \"node-resolver-9dqlz\" (UID: \"2ec19290-25c4-4b52-8173-22c14e7c9056\") " pod="openshift-dns/node-resolver-9dqlz"
Oct 30 22:06:15.902736 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.902683    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-cni-netd\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:15.902736 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.902716    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-kubelet\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:15.902902 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.902756    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-sys\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:15.902902 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.902798    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-tuned\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:15.902902 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.902832    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bdc6308d-a259-45db-90ea-094a8d1dd3a4-cni-binary-copy\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:15.902902 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.902858    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wthz4\" (UniqueName: \"kubernetes.io/projected/d303600a-2390-4ba9-8ea8-a58bd505a109-kube-api-access-wthz4\") pod \"network-check-target-zcl2w\" (UID: \"d303600a-2390-4ba9-8ea8-a58bd505a109\") " pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:15.902902 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.902891    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/47dc8b20-b520-4f4d-bd48-74b78ba300ef-env-overrides\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:15.903118 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.902925    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-run-systemd\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:15.903118 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.902978    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-etc-openvswitch\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:15.903118 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.903007    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-run-k8s-cni-cncf-io\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:15.903118 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.903053    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bdc6308d-a259-45db-90ea-094a8d1dd3a4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:15.903118 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.903081    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-var-lib-kubelet\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:15.903363 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.903123    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-hostroot\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:15.903363 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.903150    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k4xc\" (UniqueName: \"kubernetes.io/projected/2ec19290-25c4-4b52-8173-22c14e7c9056-kube-api-access-5k4xc\") pod \"node-resolver-9dqlz\" (UID: \"2ec19290-25c4-4b52-8173-22c14e7c9056\") " pod="openshift-dns/node-resolver-9dqlz"
Oct 30 22:06:15.903363 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.903175    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/47dc8b20-b520-4f4d-bd48-74b78ba300ef-ovnkube-config\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:15.903363 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.903196    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:15.903363 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.903332    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-etc-selinux\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:15.903563 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.903373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-var-lib-openvswitch\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:15.903563 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.903407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-run-openvswitch\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:15.903563 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.903425    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-run-ovn-kubernetes\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:15.903563 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.903468    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-modprobe-d\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:15.903563 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.903507    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-var-lib-cni-multus\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:15.903563 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.903555    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-systemd-units\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:15.903799 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.903595    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-run-netns\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:15.903799 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.903622    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4wvm\" (UniqueName: \"kubernetes.io/projected/10d1cecc-9e30-4354-8b42-60ddceadff44-kube-api-access-b4wvm\") pod \"node-ca-nwtzc\" (UID: \"10d1cecc-9e30-4354-8b42-60ddceadff44\") " pod="openshift-image-registry/node-ca-nwtzc"
Oct 30 22:06:15.903799 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.903651    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e444015-5346-4106-93aa-ef43eb9d0be6-dbus\") pod \"global-pull-secret-syncer-ljnhp\" (UID: \"1e444015-5346-4106-93aa-ef43eb9d0be6\") " pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:15.903799 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.903687    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-sysctl-d\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:15.903799 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.903755    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrsqc\" (UniqueName: \"kubernetes.io/projected/801d7671-e004-4da3-898e-664eb2f1b5c3-kube-api-access-jrsqc\") pod \"iptables-alerter-86vll\" (UID: \"801d7671-e004-4da3-898e-664eb2f1b5c3\") " pod="openshift-network-operator/iptables-alerter-86vll"
Oct 30 22:06:15.904014 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.903856    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-log-socket\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:15.904014 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.903883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdc6308d-a259-45db-90ea-094a8d1dd3a4-system-cni-dir\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:15.904014 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.903933    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bdc6308d-a259-45db-90ea-094a8d1dd3a4-cnibin\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:15.904014 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.903967    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bdc6308d-a259-45db-90ea-094a8d1dd3a4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:15.904014 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.904008    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-system-cni-dir\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:15.904283 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.904034    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-run-multus-certs\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:15.904283 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.904066    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/801d7671-e004-4da3-898e-664eb2f1b5c3-host-slash\") pod \"iptables-alerter-86vll\" (UID: \"801d7671-e004-4da3-898e-664eb2f1b5c3\") " pod="openshift-network-operator/iptables-alerter-86vll"
Oct 30 22:06:15.904283 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.904110    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9d1eecff-b22f-4864-a121-9fb507e783f2-agent-certs\") pod \"konnectivity-agent-2fhd5\" (UID: \"9d1eecff-b22f-4864-a121-9fb507e783f2\") " pod="kube-system/konnectivity-agent-2fhd5"
Oct 30 22:06:15.904283 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.904132    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-kubernetes\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:15.904283 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.904146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-multus-daemon-config\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:15.904283 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.904174    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e444015-5346-4106-93aa-ef43eb9d0be6-original-pull-secret\") pod \"global-pull-secret-syncer-ljnhp\" (UID: \"1e444015-5346-4106-93aa-ef43eb9d0be6\") " pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:15.904283 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.904197    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-sysconfig\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:15.904283 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.904211    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45sk7\" (UniqueName: \"kubernetes.io/projected/ccf49970-fa36-4188-ab63-e14e440b73ec-kube-api-access-45sk7\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:15.904283 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.904271    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7fw6\" (UniqueName: \"kubernetes.io/projected/4049e53b-bffe-40fb-beef-22acc171315a-kube-api-access-c7fw6\") pod \"network-metrics-daemon-fvsmv\" (UID: \"4049e53b-bffe-40fb-beef-22acc171315a\") " pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:15.904677 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.904300    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec19290-25c4-4b52-8173-22c14e7c9056-tmp-dir\") pod \"node-resolver-9dqlz\" (UID: \"2ec19290-25c4-4b52-8173-22c14e7c9056\") " pod="openshift-dns/node-resolver-9dqlz"
Oct 30 22:06:15.904677 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.904324    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/801d7671-e004-4da3-898e-664eb2f1b5c3-iptables-alerter-script\") pod \"iptables-alerter-86vll\" (UID: \"801d7671-e004-4da3-898e-664eb2f1b5c3\") " pod="openshift-network-operator/iptables-alerter-86vll"
Oct 30 22:06:15.904677 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.904344    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-socket-dir\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:15.904677 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.904359    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-node-log\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:15.904677 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.904377    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lztwd\" (UniqueName: \"kubernetes.io/projected/bdc6308d-a259-45db-90ea-094a8d1dd3a4-kube-api-access-lztwd\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:15.904677 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.904395    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-multus-cni-dir\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:15.904677 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.904420    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-cni-binary-copy\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:15.904677 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.904441    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8ltk\" (UniqueName: \"kubernetes.io/projected/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-kube-api-access-k8ltk\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:15.904677 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.904545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-sysctl-conf\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:15.904677 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.904566    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-run\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:15.904677 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.904581    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-multus-conf-dir\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:15.904677 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.904607    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e444015-5346-4106-93aa-ef43eb9d0be6-kubelet-config\") pod \"global-pull-secret-syncer-ljnhp\" (UID: \"1e444015-5346-4106-93aa-ef43eb9d0be6\") " pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:15.904677 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.904626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-cnibin\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:15.923407 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.923387    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 22:06:15.923550 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.923458    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 22:06:15.923808 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.923777    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 22:06:15.923911 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.923880    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 22:06:15.924012 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.923997    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 22:06:15.924092 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.924082    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:06:15.924148 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.924099    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 22:06:15.924148 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.924141    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 22:06:15.924356 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.924174    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:06:15.924356 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.924276    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8wxtm\""
Oct 30 22:06:15.924462 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.924444    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 22:06:15.924510 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.924486    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 22:06:15.924573 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.924556    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 22:06:15.924638 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.924618    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k6z4j\""
Oct 30 22:06:15.924690 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.924652    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 22:06:15.924736 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.924709    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g5ww4\""
Oct 30 22:06:15.924778 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.924764    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 22:06:15.924858 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.924845    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 22:06:15.924929 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.924893    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 22:06:15.924979 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.924933    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 22:06:15.924979 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.924964    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 22:06:15.925061 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.925007    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 22:06:15.925061 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.925053    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 22:06:15.936927 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.936903    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 22:06:15.937025 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.936943    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 22:06:15.937572 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.937464    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 22:06:15.937572 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.937479    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tjmjw\""
Oct 30 22:06:15.937572 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.937465    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-44l7l\""
Oct 30 22:06:15.937572 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.937532    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7kw8g\""
Oct 30 22:06:15.937572 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.937541    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 22:06:15.937572 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.937534    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xwfpf\""
Oct 30 22:06:15.937824 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.937637    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 22:06:15.937824 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:15.937646    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pwrc4\""
Oct 30 22:06:16.005270 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005236    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfdrz\" (UniqueName: \"kubernetes.io/projected/dd15a503-2539-4458-90cb-dae3e0e28e62-kube-api-access-kfdrz\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:16.005445 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005283    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-run-ovn\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.005445 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005305    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/47dc8b20-b520-4f4d-bd48-74b78ba300ef-ovnkube-script-lib\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.005445 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005331    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/10d1cecc-9e30-4354-8b42-60ddceadff44-serviceca\") pod \"node-ca-nwtzc\" (UID: \"10d1cecc-9e30-4354-8b42-60ddceadff44\") " pod="openshift-image-registry/node-ca-nwtzc"
Oct 30 22:06:16.005445 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005350    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-var-lib-kubelet\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.005445 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/47dc8b20-b520-4f4d-bd48-74b78ba300ef-ovn-node-metrics-cert\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.005445 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005412    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-run-ovn\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.005445 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005420    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-registration-dir\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:16.005758 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005468    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-device-dir\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:16.005758 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005476    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-registration-dir\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:16.005758 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005492    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-slash\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.005758 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005516    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccf49970-fa36-4188-ab63-e14e440b73ec-tmp\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.005758 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005538    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-etc-kubernetes\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.005758 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005553    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-device-dir\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:16.005758 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005559    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs\") pod \"network-metrics-daemon-fvsmv\" (UID: \"4049e53b-bffe-40fb-beef-22acc171315a\") " pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:16.005758 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005593    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bdc6308d-a259-45db-90ea-094a8d1dd3a4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:16.005758 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:16.005637    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:16.005758 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-run-netns\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.005758 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005663    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-var-lib-cni-bin\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.005758 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:16.005697    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs podName:4049e53b-bffe-40fb-beef-22acc171315a nodeName:}" failed. No retries permitted until 2026-10-30 22:06:16.505678192 +0000 UTC m=+3.075771486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs") pod "network-metrics-daemon-fvsmv" (UID: "4049e53b-bffe-40fb-beef-22acc171315a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:16.005758 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005713    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-lib-modules\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.005758 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005737    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-host\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.005758 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005759    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2ec19290-25c4-4b52-8173-22c14e7c9056-hosts-file\") pod \"node-resolver-9dqlz\" (UID: \"2ec19290-25c4-4b52-8173-22c14e7c9056\") " pod="openshift-dns/node-resolver-9dqlz"
Oct 30 22:06:16.006430 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005780    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-cni-netd\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.006430 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005802    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-kubelet\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.006430 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005822    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-sys\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.006430 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005822    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bdc6308d-a259-45db-90ea-094a8d1dd3a4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:16.006430 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005853    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-run-netns\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.006430 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005873    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-slash\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.006430 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-tuned\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.006430 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005908    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bdc6308d-a259-45db-90ea-094a8d1dd3a4-cni-binary-copy\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:16.006430 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005933    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wthz4\" (UniqueName: \"kubernetes.io/projected/d303600a-2390-4ba9-8ea8-a58bd505a109-kube-api-access-wthz4\") pod \"network-check-target-zcl2w\" (UID: \"d303600a-2390-4ba9-8ea8-a58bd505a109\") " pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:16.006430 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005956    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/47dc8b20-b520-4f4d-bd48-74b78ba300ef-env-overrides\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.006430 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005976    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-run-systemd\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.006430 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-etc-openvswitch\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.006430 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006047    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/47dc8b20-b520-4f4d-bd48-74b78ba300ef-ovnkube-script-lib\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.006430 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-run-k8s-cni-cncf-io\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.006430 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006080    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bdc6308d-a259-45db-90ea-094a8d1dd3a4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:16.006430 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006104    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-var-lib-kubelet\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.006430 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006146    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-hostroot\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.006430 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006164    2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 22:06:16.007188 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006168    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k4xc\" (UniqueName: \"kubernetes.io/projected/2ec19290-25c4-4b52-8173-22c14e7c9056-kube-api-access-5k4xc\") pod \"node-resolver-9dqlz\" (UID: \"2ec19290-25c4-4b52-8173-22c14e7c9056\") " pod="openshift-dns/node-resolver-9dqlz"
Oct 30 22:06:16.007188 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006333    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/47dc8b20-b520-4f4d-bd48-74b78ba300ef-ovnkube-config\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.007188 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006369    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:16.007188 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006394    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-etc-selinux\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:16.007188 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006417    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-var-lib-openvswitch\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.007188 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-run-openvswitch\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.007188 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006464    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-run-ovn-kubernetes\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.007188 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006471    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/10d1cecc-9e30-4354-8b42-60ddceadff44-serviceca\") pod \"node-ca-nwtzc\" (UID: \"10d1cecc-9e30-4354-8b42-60ddceadff44\") " pod="openshift-image-registry/node-ca-nwtzc"
Oct 30 22:06:16.007188 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006503    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-modprobe-d\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.007188 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006520    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-var-lib-kubelet\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.007188 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006530    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-var-lib-cni-multus\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.007188 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006547    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-etc-kubernetes\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.007188 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-systemd-units\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.007188 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006588    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-var-lib-cni-multus\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.007188 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006604    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-run-netns\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.007188 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006629    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4wvm\" (UniqueName: \"kubernetes.io/projected/10d1cecc-9e30-4354-8b42-60ddceadff44-kube-api-access-b4wvm\") pod \"node-ca-nwtzc\" (UID: \"10d1cecc-9e30-4354-8b42-60ddceadff44\") " pod="openshift-image-registry/node-ca-nwtzc"
Oct 30 22:06:16.007188 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006635    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-sys\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.007937 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006653    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e444015-5346-4106-93aa-ef43eb9d0be6-dbus\") pod \"global-pull-secret-syncer-ljnhp\" (UID: \"1e444015-5346-4106-93aa-ef43eb9d0be6\") " pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:16.007937 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006676    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-sysctl-d\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.007937 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006703    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrsqc\" (UniqueName: \"kubernetes.io/projected/801d7671-e004-4da3-898e-664eb2f1b5c3-kube-api-access-jrsqc\") pod \"iptables-alerter-86vll\" (UID: \"801d7671-e004-4da3-898e-664eb2f1b5c3\") " pod="openshift-network-operator/iptables-alerter-86vll"
Oct 30 22:06:16.007937 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006739    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-log-socket\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.007937 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006762    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdc6308d-a259-45db-90ea-094a8d1dd3a4-system-cni-dir\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:16.007937 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006785    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bdc6308d-a259-45db-90ea-094a8d1dd3a4-cnibin\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:16.007937 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.005708    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-var-lib-cni-bin\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.007937 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bdc6308d-a259-45db-90ea-094a8d1dd3a4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:16.007937 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006833    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-system-cni-dir\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.007937 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006840    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-lib-modules\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.007937 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006855    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-run-multus-certs\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.007937 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006860    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-host\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.007937 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006879    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/801d7671-e004-4da3-898e-664eb2f1b5c3-host-slash\") pod \"iptables-alerter-86vll\" (UID: \"801d7671-e004-4da3-898e-664eb2f1b5c3\") " pod="openshift-network-operator/iptables-alerter-86vll"
Oct 30 22:06:16.007937 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006888    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-run-openvswitch\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.007937 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9d1eecff-b22f-4864-a121-9fb507e783f2-agent-certs\") pod \"konnectivity-agent-2fhd5\" (UID: \"9d1eecff-b22f-4864-a121-9fb507e783f2\") " pod="kube-system/konnectivity-agent-2fhd5"
Oct 30 22:06:16.007937 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006909    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-run-netns\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.007937 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006912    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-sysctl-d\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.008679 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006920    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:16.008679 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006926    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-kubernetes\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.008679 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006949    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-cni-netd\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.008679 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006957    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-systemd-units\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.008679 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006962    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-multus-daemon-config\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.008679 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.006983    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-etc-selinux\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:16.008679 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007004    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-kubelet\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.008679 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007017    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-var-lib-openvswitch\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.008679 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007022    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-modprobe-d\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.008679 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007066    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-system-cni-dir\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.008679 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007116    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e444015-5346-4106-93aa-ef43eb9d0be6-dbus\") pod \"global-pull-secret-syncer-ljnhp\" (UID: \"1e444015-5346-4106-93aa-ef43eb9d0be6\") " pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:16.008679 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007180    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-run-multus-certs\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.008679 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007195    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-run-systemd\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.008679 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007239    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/801d7671-e004-4da3-898e-664eb2f1b5c3-host-slash\") pod \"iptables-alerter-86vll\" (UID: \"801d7671-e004-4da3-898e-664eb2f1b5c3\") " pod="openshift-network-operator/iptables-alerter-86vll"
Oct 30 22:06:16.008679 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007267    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bdc6308d-a259-45db-90ea-094a8d1dd3a4-cnibin\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:16.008679 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007271    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-kubernetes\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.008679 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007309    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-log-socket\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.008679 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007310    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-run-ovn-kubernetes\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.009657 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007356    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdc6308d-a259-45db-90ea-094a8d1dd3a4-system-cni-dir\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:16.009657 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007364    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-run-k8s-cni-cncf-io\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.009657 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007415    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2ec19290-25c4-4b52-8173-22c14e7c9056-hosts-file\") pod \"node-resolver-9dqlz\" (UID: \"2ec19290-25c4-4b52-8173-22c14e7c9056\") " pod="openshift-dns/node-resolver-9dqlz"
Oct 30 22:06:16.009657 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007455    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-hostroot\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.009657 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007587    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-etc-openvswitch\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.009657 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007595    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-var-lib-kubelet\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.009657 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007628    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e444015-5346-4106-93aa-ef43eb9d0be6-original-pull-secret\") pod \"global-pull-secret-syncer-ljnhp\" (UID: \"1e444015-5346-4106-93aa-ef43eb9d0be6\") " pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:16.009657 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007653    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-sysconfig\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.009657 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45sk7\" (UniqueName: \"kubernetes.io/projected/ccf49970-fa36-4188-ab63-e14e440b73ec-kube-api-access-45sk7\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.009657 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7fw6\" (UniqueName: \"kubernetes.io/projected/4049e53b-bffe-40fb-beef-22acc171315a-kube-api-access-c7fw6\") pod \"network-metrics-daemon-fvsmv\" (UID: \"4049e53b-bffe-40fb-beef-22acc171315a\") " pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:16.009657 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:16.007715    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:16.009657 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:16.007763    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e444015-5346-4106-93aa-ef43eb9d0be6-original-pull-secret podName:1e444015-5346-4106-93aa-ef43eb9d0be6 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:16.507748253 +0000 UTC m=+3.077841537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e444015-5346-4106-93aa-ef43eb9d0be6-original-pull-secret") pod "global-pull-secret-syncer-ljnhp" (UID: "1e444015-5346-4106-93aa-ef43eb9d0be6") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:16.009657 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007716    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec19290-25c4-4b52-8173-22c14e7c9056-tmp-dir\") pod \"node-resolver-9dqlz\" (UID: \"2ec19290-25c4-4b52-8173-22c14e7c9056\") " pod="openshift-dns/node-resolver-9dqlz"
Oct 30 22:06:16.009657 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/801d7671-e004-4da3-898e-664eb2f1b5c3-iptables-alerter-script\") pod \"iptables-alerter-86vll\" (UID: \"801d7671-e004-4da3-898e-664eb2f1b5c3\") " pod="openshift-network-operator/iptables-alerter-86vll"
Oct 30 22:06:16.009657 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-socket-dir\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:16.009657 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007849    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-node-log\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.009657 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007875    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lztwd\" (UniqueName: \"kubernetes.io/projected/bdc6308d-a259-45db-90ea-094a8d1dd3a4-kube-api-access-lztwd\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:16.010436 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007899    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-multus-cni-dir\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.010436 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007903    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-sysconfig\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.010436 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007923    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-cni-binary-copy\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.010436 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8ltk\" (UniqueName: \"kubernetes.io/projected/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-kube-api-access-k8ltk\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.010436 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007965    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-sysctl-conf\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.010436 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.007987    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-run\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.010436 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.008008    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-multus-conf-dir\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.010436 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.008033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e444015-5346-4106-93aa-ef43eb9d0be6-kubelet-config\") pod \"global-pull-secret-syncer-ljnhp\" (UID: \"1e444015-5346-4106-93aa-ef43eb9d0be6\") " pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:16.010436 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.008055    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-cnibin\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.010436 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.008080    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.010436 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.008104    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swswj\" (UniqueName: \"kubernetes.io/projected/47dc8b20-b520-4f4d-bd48-74b78ba300ef-kube-api-access-swswj\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.010436 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.008126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9d1eecff-b22f-4864-a121-9fb507e783f2-konnectivity-ca\") pod \"konnectivity-agent-2fhd5\" (UID: \"9d1eecff-b22f-4864-a121-9fb507e783f2\") " pod="kube-system/konnectivity-agent-2fhd5"
Oct 30 22:06:16.010436 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.008147    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10d1cecc-9e30-4354-8b42-60ddceadff44-host\") pod \"node-ca-nwtzc\" (UID: \"10d1cecc-9e30-4354-8b42-60ddceadff44\") " pod="openshift-image-registry/node-ca-nwtzc"
Oct 30 22:06:16.010436 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.008177    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-systemd\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.010436 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.008200    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-multus-socket-dir-parent\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.010436 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.008245    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-cni-bin\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.010436 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.008270    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-sys-fs\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:16.010436 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.008352    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-sys-fs\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:16.011262 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.008544    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/47dc8b20-b520-4f4d-bd48-74b78ba300ef-env-overrides\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.011262 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.008600    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-multus-conf-dir\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.011262 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.008631    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bdc6308d-a259-45db-90ea-094a8d1dd3a4-cni-binary-copy\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:16.011262 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.008676    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-socket-dir\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:16.011262 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.008710    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-node-log\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.011262 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.008716    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bdc6308d-a259-45db-90ea-094a8d1dd3a4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:16.011262 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.008774    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-cnibin\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.011262 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.008811    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e444015-5346-4106-93aa-ef43eb9d0be6-kubelet-config\") pod \"global-pull-secret-syncer-ljnhp\" (UID: \"1e444015-5346-4106-93aa-ef43eb9d0be6\") " pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:16.011262 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.008049    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/47dc8b20-b520-4f4d-bd48-74b78ba300ef-ovnkube-config\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.011262 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.008848    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/801d7671-e004-4da3-898e-664eb2f1b5c3-iptables-alerter-script\") pod \"iptables-alerter-86vll\" (UID: \"801d7671-e004-4da3-898e-664eb2f1b5c3\") " pod="openshift-network-operator/iptables-alerter-86vll"
Oct 30 22:06:16.011262 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.008857    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.011262 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.008917    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10d1cecc-9e30-4354-8b42-60ddceadff44-host\") pod \"node-ca-nwtzc\" (UID: \"10d1cecc-9e30-4354-8b42-60ddceadff44\") " pod="openshift-image-registry/node-ca-nwtzc"
Oct 30 22:06:16.011262 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.008961    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-sysctl-conf\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.011262 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.008974    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-systemd\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.011262 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.009022    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-multus-socket-dir-parent\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.011262 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.009051    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-run\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.011262 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.009063    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-cni-bin\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.011262 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.009109    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-multus-cni-dir\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.012007 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.009702    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec19290-25c4-4b52-8173-22c14e7c9056-tmp-dir\") pod \"node-resolver-9dqlz\" (UID: \"2ec19290-25c4-4b52-8173-22c14e7c9056\") " pod="openshift-dns/node-resolver-9dqlz"
Oct 30 22:06:16.012007 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.010065    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bdc6308d-a259-45db-90ea-094a8d1dd3a4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:16.012007 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.010179    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9d1eecff-b22f-4864-a121-9fb507e783f2-konnectivity-ca\") pod \"konnectivity-agent-2fhd5\" (UID: \"9d1eecff-b22f-4864-a121-9fb507e783f2\") " pod="kube-system/konnectivity-agent-2fhd5"
Oct 30 22:06:16.012007 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.010246    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-multus-daemon-config\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.012007 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.010273    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-cni-binary-copy\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.012268 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.012009    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/47dc8b20-b520-4f4d-bd48-74b78ba300ef-ovn-node-metrics-cert\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.012268 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.012234    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-tuned\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.012411 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.012386    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccf49970-fa36-4188-ab63-e14e440b73ec-tmp\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.014606 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.014587    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9d1eecff-b22f-4864-a121-9fb507e783f2-agent-certs\") pod \"konnectivity-agent-2fhd5\" (UID: \"9d1eecff-b22f-4864-a121-9fb507e783f2\") " pod="kube-system/konnectivity-agent-2fhd5"
Oct 30 22:06:16.026530 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:16.026503    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:06:16.026530 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:16.026533    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:06:16.026692 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:16.026545    2580 projected.go:196] Error preparing data for projected volume kube-api-access-wthz4 for pod openshift-network-diagnostics/network-check-target-zcl2w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:16.026692 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:16.026597    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d303600a-2390-4ba9-8ea8-a58bd505a109-kube-api-access-wthz4 podName:d303600a-2390-4ba9-8ea8-a58bd505a109 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:16.526584018 +0000 UTC m=+3.096677289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wthz4" (UniqueName: "kubernetes.io/projected/d303600a-2390-4ba9-8ea8-a58bd505a109-kube-api-access-wthz4") pod "network-check-target-zcl2w" (UID: "d303600a-2390-4ba9-8ea8-a58bd505a109") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:16.029242 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.029147    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfdrz\" (UniqueName: \"kubernetes.io/projected/dd15a503-2539-4458-90cb-dae3e0e28e62-kube-api-access-kfdrz\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:16.030897 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.030835    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k4xc\" (UniqueName: \"kubernetes.io/projected/2ec19290-25c4-4b52-8173-22c14e7c9056-kube-api-access-5k4xc\") pod \"node-resolver-9dqlz\" (UID: \"2ec19290-25c4-4b52-8173-22c14e7c9056\") " pod="openshift-dns/node-resolver-9dqlz"
Oct 30 22:06:16.032551 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.032516    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45sk7\" (UniqueName: \"kubernetes.io/projected/ccf49970-fa36-4188-ab63-e14e440b73ec-kube-api-access-45sk7\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.032975 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.032957    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4wvm\" (UniqueName: \"kubernetes.io/projected/10d1cecc-9e30-4354-8b42-60ddceadff44-kube-api-access-b4wvm\") pod \"node-ca-nwtzc\" (UID: \"10d1cecc-9e30-4354-8b42-60ddceadff44\") " pod="openshift-image-registry/node-ca-nwtzc"
Oct 30 22:06:16.032975 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.032967    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrsqc\" (UniqueName: \"kubernetes.io/projected/801d7671-e004-4da3-898e-664eb2f1b5c3-kube-api-access-jrsqc\") pod \"iptables-alerter-86vll\" (UID: \"801d7671-e004-4da3-898e-664eb2f1b5c3\") " pod="openshift-network-operator/iptables-alerter-86vll"
Oct 30 22:06:16.033102 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.033026    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lztwd\" (UniqueName: \"kubernetes.io/projected/bdc6308d-a259-45db-90ea-094a8d1dd3a4-kube-api-access-lztwd\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:16.033195 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.033167    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7fw6\" (UniqueName: \"kubernetes.io/projected/4049e53b-bffe-40fb-beef-22acc171315a-kube-api-access-c7fw6\") pod \"network-metrics-daemon-fvsmv\" (UID: \"4049e53b-bffe-40fb-beef-22acc171315a\") " pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:16.034879 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.034859    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swswj\" (UniqueName: \"kubernetes.io/projected/47dc8b20-b520-4f4d-bd48-74b78ba300ef-kube-api-access-swswj\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.041645 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.041625    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8ltk\" (UniqueName: \"kubernetes.io/projected/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-kube-api-access-k8ltk\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.125637 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.125567    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9dqlz"
Oct 30 22:06:16.135448 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.135281    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nwtzc"
Oct 30 22:06:16.135758 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:16.135718    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec19290_25c4_4b52_8173_22c14e7c9056.slice/crio-e639ca449fd6481bd8538d41484639a47ad2d027e5d81ec2deb1b336f6c6c617 WatchSource:0}: Error finding container e639ca449fd6481bd8538d41484639a47ad2d027e5d81ec2deb1b336f6c6c617: Status 404 returned error can't find the container with id e639ca449fd6481bd8538d41484639a47ad2d027e5d81ec2deb1b336f6c6c617
Oct 30 22:06:16.145389 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:16.145269    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10d1cecc_9e30_4354_8b42_60ddceadff44.slice/crio-3739156ea50346f67c1f1c70b6a7bd166cc518bbce04c34908a9c1148695a299 WatchSource:0}: Error finding container 3739156ea50346f67c1f1c70b6a7bd166cc518bbce04c34908a9c1148695a299: Status 404 returned error can't find the container with id 3739156ea50346f67c1f1c70b6a7bd166cc518bbce04c34908a9c1148695a299
Oct 30 22:06:16.147098 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.147083    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:16.152709 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.152688    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-86vll"
Oct 30 22:06:16.156861 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:16.156838    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf49970_fa36_4188_ab63_e14e440b73ec.slice/crio-9800b06ff5979634bdf0f80e560a396bf360393d202675f8163bcf640ca4ba93 WatchSource:0}: Error finding container 9800b06ff5979634bdf0f80e560a396bf360393d202675f8163bcf640ca4ba93: Status 404 returned error can't find the container with id 9800b06ff5979634bdf0f80e560a396bf360393d202675f8163bcf640ca4ba93
Oct 30 22:06:16.161989 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:16.161963    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801d7671_e004_4da3_898e_664eb2f1b5c3.slice/crio-20d5b724c1ef9e807e1f97ce33a81114cbb8ebca9d46488a059e263ff1a7e1ee WatchSource:0}: Error finding container 20d5b724c1ef9e807e1f97ce33a81114cbb8ebca9d46488a059e263ff1a7e1ee: Status 404 returned error can't find the container with id 20d5b724c1ef9e807e1f97ce33a81114cbb8ebca9d46488a059e263ff1a7e1ee
Oct 30 22:06:16.163779 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.163762    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:16.178267 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.178250    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:16.184523 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.184505    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:16.186649 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.186630    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2fhd5"
Oct 30 22:06:16.188945 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:16.188924    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e64d9bd_0547_4c6d_adc4_8a3ce9bc0fb9.slice/crio-03750ea778f9d46aafdb422942074845d557e3711abaffa235c54ae20950287d WatchSource:0}: Error finding container 03750ea778f9d46aafdb422942074845d557e3711abaffa235c54ae20950287d: Status 404 returned error can't find the container with id 03750ea778f9d46aafdb422942074845d557e3711abaffa235c54ae20950287d
Oct 30 22:06:16.194563 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:16.194541    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc6308d_a259_45db_90ea_094a8d1dd3a4.slice/crio-907de9fd04680f90043bccba46fefda4567401cf21b11ec468c4bc55d2492d6a WatchSource:0}: Error finding container 907de9fd04680f90043bccba46fefda4567401cf21b11ec468c4bc55d2492d6a: Status 404 returned error can't find the container with id 907de9fd04680f90043bccba46fefda4567401cf21b11ec468c4bc55d2492d6a
Oct 30 22:06:16.197134 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:16.196917    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1eecff_b22f_4864_a121_9fb507e783f2.slice/crio-f3a4ea6cd48a49887913384ad1cc06bef23dfe6263904754a20538780da8eea3 WatchSource:0}: Error finding container f3a4ea6cd48a49887913384ad1cc06bef23dfe6263904754a20538780da8eea3: Status 404 returned error can't find the container with id f3a4ea6cd48a49887913384ad1cc06bef23dfe6263904754a20538780da8eea3
Oct 30 22:06:16.202344 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.202326    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:16.228463 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:16.228441    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47dc8b20_b520_4f4d_bd48_74b78ba300ef.slice/crio-41bec762b666a037acb1960a7ddf13317cfcc5a451f2d3c7aab97a47dd7bd6e4 WatchSource:0}: Error finding container 41bec762b666a037acb1960a7ddf13317cfcc5a451f2d3c7aab97a47dd7bd6e4: Status 404 returned error can't find the container with id 41bec762b666a037acb1960a7ddf13317cfcc5a451f2d3c7aab97a47dd7bd6e4
Oct 30 22:06:16.514349 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.513296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e444015-5346-4106-93aa-ef43eb9d0be6-original-pull-secret\") pod \"global-pull-secret-syncer-ljnhp\" (UID: \"1e444015-5346-4106-93aa-ef43eb9d0be6\") " pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:16.514349 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.513567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs\") pod \"network-metrics-daemon-fvsmv\" (UID: \"4049e53b-bffe-40fb-beef-22acc171315a\") " pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:16.514349 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:16.513710    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:16.514349 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:16.513788    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs podName:4049e53b-bffe-40fb-beef-22acc171315a nodeName:}" failed. No retries permitted until 2026-10-30 22:06:17.513773173 +0000 UTC m=+4.083866457 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs") pod "network-metrics-daemon-fvsmv" (UID: "4049e53b-bffe-40fb-beef-22acc171315a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:16.514349 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:16.514258    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:16.514349 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:16.514318    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e444015-5346-4106-93aa-ef43eb9d0be6-original-pull-secret podName:1e444015-5346-4106-93aa-ef43eb9d0be6 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:17.514304079 +0000 UTC m=+4.084397367 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e444015-5346-4106-93aa-ef43eb9d0be6-original-pull-secret") pod "global-pull-secret-syncer-ljnhp" (UID: "1e444015-5346-4106-93aa-ef43eb9d0be6") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:16.614589 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.614553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wthz4\" (UniqueName: \"kubernetes.io/projected/d303600a-2390-4ba9-8ea8-a58bd505a109-kube-api-access-wthz4\") pod \"network-check-target-zcl2w\" (UID: \"d303600a-2390-4ba9-8ea8-a58bd505a109\") " pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:16.614848 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:16.614716    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:06:16.614848 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:16.614733    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:06:16.614848 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:16.614742    2580 projected.go:196] Error preparing data for projected volume kube-api-access-wthz4 for pod openshift-network-diagnostics/network-check-target-zcl2w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:16.614848 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:16.614792    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d303600a-2390-4ba9-8ea8-a58bd505a109-kube-api-access-wthz4 podName:d303600a-2390-4ba9-8ea8-a58bd505a109 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:17.614776721 +0000 UTC m=+4.184869992 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wthz4" (UniqueName: "kubernetes.io/projected/d303600a-2390-4ba9-8ea8-a58bd505a109-kube-api-access-wthz4") pod "network-check-target-zcl2w" (UID: "d303600a-2390-4ba9-8ea8-a58bd505a109") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:16.850545 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.850486    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 22:01:15 +0000 UTC" deadline="2028-04-06 19:13:37.023933389 +0000 UTC"
Oct 30 22:06:16.850545 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.850543    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12573h7m20.173393753s"
Oct 30 22:06:16.899914 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.899808    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9dqlz" event={"ID":"2ec19290-25c4-4b52-8173-22c14e7c9056","Type":"ContainerStarted","Data":"e639ca449fd6481bd8538d41484639a47ad2d027e5d81ec2deb1b336f6c6c617"}
Oct 30 22:06:16.906452 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.905983    2580 generic.go:356] "Generic (PLEG): container finished" podID="c3e2926fda086fe0743710e38cf91711" containerID="dc37d4cca2b85424065882973fbb214ed65b63b822dc095e27375a130e7fe443" exitCode=0
Oct 30 22:06:16.906452 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.906058    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal" event={"ID":"c3e2926fda086fe0743710e38cf91711","Type":"ContainerDied","Data":"dc37d4cca2b85424065882973fbb214ed65b63b822dc095e27375a130e7fe443"}
Oct 30 22:06:16.911987 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.911913    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2fhd5" event={"ID":"9d1eecff-b22f-4864-a121-9fb507e783f2","Type":"ContainerStarted","Data":"f3a4ea6cd48a49887913384ad1cc06bef23dfe6263904754a20538780da8eea3"}
Oct 30 22:06:16.919248 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.919152    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v69rv" event={"ID":"bdc6308d-a259-45db-90ea-094a8d1dd3a4","Type":"ContainerStarted","Data":"907de9fd04680f90043bccba46fefda4567401cf21b11ec468c4bc55d2492d6a"}
Oct 30 22:06:16.926541 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.926494    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p8lsb" event={"ID":"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9","Type":"ContainerStarted","Data":"03750ea778f9d46aafdb422942074845d557e3711abaffa235c54ae20950287d"}
Oct 30 22:06:16.935025 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.934993    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v" event={"ID":"dd15a503-2539-4458-90cb-dae3e0e28e62","Type":"ContainerStarted","Data":"3858b6612025b28ff98838f7ebf46418f61ea0e479908653b17a4d70c3bb3908"}
Oct 30 22:06:16.939711 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.939681    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-86vll" event={"ID":"801d7671-e004-4da3-898e-664eb2f1b5c3","Type":"ContainerStarted","Data":"20d5b724c1ef9e807e1f97ce33a81114cbb8ebca9d46488a059e263ff1a7e1ee"}
Oct 30 22:06:16.952734 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.952702    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nwtzc" event={"ID":"10d1cecc-9e30-4354-8b42-60ddceadff44","Type":"ContainerStarted","Data":"3739156ea50346f67c1f1c70b6a7bd166cc518bbce04c34908a9c1148695a299"}
Oct 30 22:06:16.964564 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.964157    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-185.ec2.internal" event={"ID":"496cd66cfb3bc1cef71f828d2683a76e","Type":"ContainerStarted","Data":"050605e2bb6fc90d8898833cc2f8ea6c6261849a2bdd0d0e57e1264527992606"}
Oct 30 22:06:16.982905 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:16.982874    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5" event={"ID":"47dc8b20-b520-4f4d-bd48-74b78ba300ef","Type":"ContainerStarted","Data":"41bec762b666a037acb1960a7ddf13317cfcc5a451f2d3c7aab97a47dd7bd6e4"}
Oct 30 22:06:17.001184 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:17.001130    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-185.ec2.internal" podStartSLOduration=3.001113872 podStartE2EDuration="3.001113872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:17.001064273 +0000 UTC m=+3.571157586" watchObservedRunningTime="2026-10-30 22:06:17.001113872 +0000 UTC m=+3.571207186"
Oct 30 22:06:17.007635 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:17.007559    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p4g27" event={"ID":"ccf49970-fa36-4188-ab63-e14e440b73ec","Type":"ContainerStarted","Data":"9800b06ff5979634bdf0f80e560a396bf360393d202675f8163bcf640ca4ba93"}
Oct 30 22:06:17.520617 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:17.520584    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e444015-5346-4106-93aa-ef43eb9d0be6-original-pull-secret\") pod \"global-pull-secret-syncer-ljnhp\" (UID: \"1e444015-5346-4106-93aa-ef43eb9d0be6\") " pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:17.520781 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:17.520648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs\") pod \"network-metrics-daemon-fvsmv\" (UID: \"4049e53b-bffe-40fb-beef-22acc171315a\") " pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:17.520781 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:17.520776    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:17.520914 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:17.520841    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs podName:4049e53b-bffe-40fb-beef-22acc171315a nodeName:}" failed. No retries permitted until 2026-10-30 22:06:19.520815451 +0000 UTC m=+6.090908735 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs") pod "network-metrics-daemon-fvsmv" (UID: "4049e53b-bffe-40fb-beef-22acc171315a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:17.521205 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:17.521188    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:17.521329 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:17.521258    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e444015-5346-4106-93aa-ef43eb9d0be6-original-pull-secret podName:1e444015-5346-4106-93aa-ef43eb9d0be6 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:19.521242943 +0000 UTC m=+6.091336219 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e444015-5346-4106-93aa-ef43eb9d0be6-original-pull-secret") pod "global-pull-secret-syncer-ljnhp" (UID: "1e444015-5346-4106-93aa-ef43eb9d0be6") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:17.621856 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:17.621820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wthz4\" (UniqueName: \"kubernetes.io/projected/d303600a-2390-4ba9-8ea8-a58bd505a109-kube-api-access-wthz4\") pod \"network-check-target-zcl2w\" (UID: \"d303600a-2390-4ba9-8ea8-a58bd505a109\") " pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:17.622035 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:17.622016    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:06:17.622035 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:17.622034    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:06:17.622174 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:17.622047    2580 projected.go:196] Error preparing data for projected volume kube-api-access-wthz4 for pod openshift-network-diagnostics/network-check-target-zcl2w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:17.622174 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:17.622100    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d303600a-2390-4ba9-8ea8-a58bd505a109-kube-api-access-wthz4 podName:d303600a-2390-4ba9-8ea8-a58bd505a109 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:19.622083905 +0000 UTC m=+6.192177250 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wthz4" (UniqueName: "kubernetes.io/projected/d303600a-2390-4ba9-8ea8-a58bd505a109-kube-api-access-wthz4") pod "network-check-target-zcl2w" (UID: "d303600a-2390-4ba9-8ea8-a58bd505a109") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:17.851685 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:17.851582    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 22:01:15 +0000 UTC" deadline="2028-06-26 07:32:43.841519684 +0000 UTC"
Oct 30 22:06:17.851685 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:17.851641    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14505h26m25.989883044s"
Oct 30 22:06:17.874464 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:17.874349    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:17.874464 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:17.874379    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:17.874687 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:17.874468    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljnhp" podUID="1e444015-5346-4106-93aa-ef43eb9d0be6"
Oct 30 22:06:17.875047 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:17.874799    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:17.875047 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:17.874904    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcl2w" podUID="d303600a-2390-4ba9-8ea8-a58bd505a109"
Oct 30 22:06:17.875047 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:17.875003    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvsmv" podUID="4049e53b-bffe-40fb-beef-22acc171315a"
Oct 30 22:06:18.030301 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:18.030272    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal" event={"ID":"c3e2926fda086fe0743710e38cf91711","Type":"ContainerStarted","Data":"77c95add65b87af616cd566bbbf5da6ddc3d60cb5bdc96ee8d2382bb9dc1cf8b"}
Oct 30 22:06:19.037196 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:19.035960    2580 generic.go:356] "Generic (PLEG): container finished" podID="bdc6308d-a259-45db-90ea-094a8d1dd3a4" containerID="172e6bd2a08937872132948b1506e962b0bbfb589b773266647184426126fea8" exitCode=0
Oct 30 22:06:19.037196 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:19.036946    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v69rv" event={"ID":"bdc6308d-a259-45db-90ea-094a8d1dd3a4","Type":"ContainerDied","Data":"172e6bd2a08937872132948b1506e962b0bbfb589b773266647184426126fea8"}
Oct 30 22:06:19.106955 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:19.106592    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal" podStartSLOduration=5.10657169 podStartE2EDuration="5.10657169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:18.070067014 +0000 UTC m=+4.640160308" watchObservedRunningTime="2026-10-30 22:06:19.10657169 +0000 UTC m=+5.676664982"
Oct 30 22:06:19.536031 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:19.535996    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e444015-5346-4106-93aa-ef43eb9d0be6-original-pull-secret\") pod \"global-pull-secret-syncer-ljnhp\" (UID: \"1e444015-5346-4106-93aa-ef43eb9d0be6\") " pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:19.536192 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:19.536060    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs\") pod \"network-metrics-daemon-fvsmv\" (UID: \"4049e53b-bffe-40fb-beef-22acc171315a\") " pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:19.536192 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:19.536178    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:19.536314 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:19.536257    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs podName:4049e53b-bffe-40fb-beef-22acc171315a nodeName:}" failed. No retries permitted until 2026-10-30 22:06:23.536240292 +0000 UTC m=+10.106333576 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs") pod "network-metrics-daemon-fvsmv" (UID: "4049e53b-bffe-40fb-beef-22acc171315a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:19.536383 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:19.536315    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:19.536383 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:19.536345    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e444015-5346-4106-93aa-ef43eb9d0be6-original-pull-secret podName:1e444015-5346-4106-93aa-ef43eb9d0be6 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:23.536335858 +0000 UTC m=+10.106429132 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e444015-5346-4106-93aa-ef43eb9d0be6-original-pull-secret") pod "global-pull-secret-syncer-ljnhp" (UID: "1e444015-5346-4106-93aa-ef43eb9d0be6") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:19.637134 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:19.637096    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wthz4\" (UniqueName: \"kubernetes.io/projected/d303600a-2390-4ba9-8ea8-a58bd505a109-kube-api-access-wthz4\") pod \"network-check-target-zcl2w\" (UID: \"d303600a-2390-4ba9-8ea8-a58bd505a109\") " pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:19.637316 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:19.637300    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:06:19.637378 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:19.637321    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:06:19.637378 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:19.637333    2580 projected.go:196] Error preparing data for projected volume kube-api-access-wthz4 for pod openshift-network-diagnostics/network-check-target-zcl2w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:19.637471 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:19.637446    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d303600a-2390-4ba9-8ea8-a58bd505a109-kube-api-access-wthz4 podName:d303600a-2390-4ba9-8ea8-a58bd505a109 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:23.63741166 +0000 UTC m=+10.207504937 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wthz4" (UniqueName: "kubernetes.io/projected/d303600a-2390-4ba9-8ea8-a58bd505a109-kube-api-access-wthz4") pod "network-check-target-zcl2w" (UID: "d303600a-2390-4ba9-8ea8-a58bd505a109") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:19.875357 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:19.875329    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:19.875488 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:19.875464    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvsmv" podUID="4049e53b-bffe-40fb-beef-22acc171315a"
Oct 30 22:06:19.875569 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:19.875528    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:19.875623 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:19.875603    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcl2w" podUID="d303600a-2390-4ba9-8ea8-a58bd505a109"
Oct 30 22:06:19.876172 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:19.876154    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:19.876290 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:19.876265    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljnhp" podUID="1e444015-5346-4106-93aa-ef43eb9d0be6"
Oct 30 22:06:20.042695 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:20.042654    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2fhd5" event={"ID":"9d1eecff-b22f-4864-a121-9fb507e783f2","Type":"ContainerStarted","Data":"09f6d1f2be8edf54631ad39805c7f9ff4dce5e99ab195a7bb1147d7ad4731cfe"}
Oct 30 22:06:20.087016 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:20.086238    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2fhd5" podStartSLOduration=3.617512361 podStartE2EDuration="7.086200204s" totalImagesPullingTime="3.468687843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:20.085648716 +0000 UTC m=+6.655742012" watchObservedRunningTime="2026-10-30 22:06:20.086200204 +0000 UTC m=+6.656293512"
Oct 30 22:06:21.049262 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:21.048949    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nwtzc" event={"ID":"10d1cecc-9e30-4354-8b42-60ddceadff44","Type":"ContainerStarted","Data":"deec2b164503ddc48db2fe1e1face6f24257b4441295e03ff186fd3cbeedfb3f"}
Oct 30 22:06:21.876424 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:21.875652    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:21.876424 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:21.875788    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljnhp" podUID="1e444015-5346-4106-93aa-ef43eb9d0be6"
Oct 30 22:06:21.876424 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:21.876143    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:21.876424 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:21.876242    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvsmv" podUID="4049e53b-bffe-40fb-beef-22acc171315a"
Oct 30 22:06:21.876424 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:21.876296    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:21.876424 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:21.876361    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcl2w" podUID="d303600a-2390-4ba9-8ea8-a58bd505a109"
Oct 30 22:06:22.055514 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:22.055477    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v" event={"ID":"dd15a503-2539-4458-90cb-dae3e0e28e62","Type":"ContainerStarted","Data":"e909cb04727bb801902f66f65128e05986b8b17de59487974ed25e4a0027ca26"}
Oct 30 22:06:22.728809 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:22.728738    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2fhd5"
Oct 30 22:06:22.729861 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:22.729841    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2fhd5"
Oct 30 22:06:22.773040 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:22.772804    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nwtzc" podStartSLOduration=5.113776425 podStartE2EDuration="9.772784103s" totalImagesPullingTime="4.659007678s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:21.09174206 +0000 UTC m=+7.661835355" watchObservedRunningTime="2026-10-30 22:06:22.772784103 +0000 UTC m=+9.342877401"
Oct 30 22:06:22.994084 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:22.993887    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2fhd5"
Oct 30 22:06:22.994707 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:22.994507    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2fhd5"
Oct 30 22:06:23.364134 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:23.364098    2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 22:06:23.575614 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:23.575493    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e444015-5346-4106-93aa-ef43eb9d0be6-original-pull-secret\") pod \"global-pull-secret-syncer-ljnhp\" (UID: \"1e444015-5346-4106-93aa-ef43eb9d0be6\") " pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:23.575614 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:23.575574    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs\") pod \"network-metrics-daemon-fvsmv\" (UID: \"4049e53b-bffe-40fb-beef-22acc171315a\") " pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:23.575836 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:23.575659    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:23.575836 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:23.575700    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:23.575836 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:23.575736    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e444015-5346-4106-93aa-ef43eb9d0be6-original-pull-secret podName:1e444015-5346-4106-93aa-ef43eb9d0be6 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:31.575715322 +0000 UTC m=+18.145808635 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e444015-5346-4106-93aa-ef43eb9d0be6-original-pull-secret") pod "global-pull-secret-syncer-ljnhp" (UID: "1e444015-5346-4106-93aa-ef43eb9d0be6") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:23.575836 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:23.575759    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs podName:4049e53b-bffe-40fb-beef-22acc171315a nodeName:}" failed. No retries permitted until 2026-10-30 22:06:31.575747847 +0000 UTC m=+18.145841121 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs") pod "network-metrics-daemon-fvsmv" (UID: "4049e53b-bffe-40fb-beef-22acc171315a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:23.676133 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:23.676109    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wthz4\" (UniqueName: \"kubernetes.io/projected/d303600a-2390-4ba9-8ea8-a58bd505a109-kube-api-access-wthz4\") pod \"network-check-target-zcl2w\" (UID: \"d303600a-2390-4ba9-8ea8-a58bd505a109\") " pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:23.676311 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:23.676274    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:06:23.676311 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:23.676293    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:06:23.676311 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:23.676305    2580 projected.go:196] Error preparing data for projected volume kube-api-access-wthz4 for pod openshift-network-diagnostics/network-check-target-zcl2w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:23.676462 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:23.676390    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d303600a-2390-4ba9-8ea8-a58bd505a109-kube-api-access-wthz4 podName:d303600a-2390-4ba9-8ea8-a58bd505a109 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:31.676372508 +0000 UTC m=+18.246465795 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wthz4" (UniqueName: "kubernetes.io/projected/d303600a-2390-4ba9-8ea8-a58bd505a109-kube-api-access-wthz4") pod "network-check-target-zcl2w" (UID: "d303600a-2390-4ba9-8ea8-a58bd505a109") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:23.867750 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:23.866998    2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T22:06:23.36412801Z","UUID":"e8a912e2-45c2-4078-8404-4817fac06952","Handler":null,"Name":"","Endpoint":""}
Oct 30 22:06:23.868797 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:23.868773    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 22:06:23.868797 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:23.868799    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 22:06:23.876398 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:23.876366    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:23.876549 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:23.876489    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcl2w" podUID="d303600a-2390-4ba9-8ea8-a58bd505a109"
Oct 30 22:06:23.876898 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:23.876876    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:23.876997 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:23.876977    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljnhp" podUID="1e444015-5346-4106-93aa-ef43eb9d0be6"
Oct 30 22:06:23.877064 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:23.877039    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:23.877134 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:23.877116    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvsmv" podUID="4049e53b-bffe-40fb-beef-22acc171315a"
Oct 30 22:06:24.062028 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:24.061522    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9dqlz" event={"ID":"2ec19290-25c4-4b52-8173-22c14e7c9056","Type":"ContainerStarted","Data":"a9b0303d08a4e906882130018e32f93f207e09479ce88f4ca5392fef6b8e25d6"}
Oct 30 22:06:24.064114 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:24.064080    2580 generic.go:356] "Generic (PLEG): container finished" podID="bdc6308d-a259-45db-90ea-094a8d1dd3a4" containerID="13789b38308d9965add9531d538c51cafadcc52906304558f308b7084b3c5ffc" exitCode=0
Oct 30 22:06:24.064262 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:24.064162    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v69rv" event={"ID":"bdc6308d-a259-45db-90ea-094a8d1dd3a4","Type":"ContainerDied","Data":"13789b38308d9965add9531d538c51cafadcc52906304558f308b7084b3c5ffc"}
Oct 30 22:06:24.067299 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:24.067272    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p8lsb" event={"ID":"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9","Type":"ContainerStarted","Data":"85b7e41d31d56c141fe3eaa38de5aa8187434764cf4031de3d544d48db441a75"}
Oct 30 22:06:24.070534 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:24.070510    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v" event={"ID":"dd15a503-2539-4458-90cb-dae3e0e28e62","Type":"ContainerStarted","Data":"5aac734b81c16fcecf648ed2f815d7c5b85de95be18b505310165e6f540ca61a"}
Oct 30 22:06:24.151942 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:24.151884    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-9dqlz" podStartSLOduration=3.592133958 podStartE2EDuration="11.15186443s" totalImagesPullingTime="7.559730472s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:24.104400071 +0000 UTC m=+10.674493370" watchObservedRunningTime="2026-10-30 22:06:24.15186443 +0000 UTC m=+10.721957728"
Oct 30 22:06:25.075439 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:25.075181    2580 generic.go:356] "Generic (PLEG): container finished" podID="bdc6308d-a259-45db-90ea-094a8d1dd3a4" containerID="bf22235e056f0bc95aff5bafa0dc1df8420e1cf1182ae2a5089e6fc9ce8cf198" exitCode=0
Oct 30 22:06:25.075900 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:25.075324    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v69rv" event={"ID":"bdc6308d-a259-45db-90ea-094a8d1dd3a4","Type":"ContainerDied","Data":"bf22235e056f0bc95aff5bafa0dc1df8420e1cf1182ae2a5089e6fc9ce8cf198"}
Oct 30 22:06:25.078375 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:25.078343    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v" event={"ID":"dd15a503-2539-4458-90cb-dae3e0e28e62","Type":"ContainerStarted","Data":"83b4c0d5f4dffa157c5f302fee5c20933b9f4358803df4f749741d68bf15418f"}
Oct 30 22:06:25.123972 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:25.123912    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-p8lsb" podStartSLOduration=4.629236372 podStartE2EDuration="12.123897868s" totalImagesPullingTime="7.494661496s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:24.194464312 +0000 UTC m=+10.764557610" watchObservedRunningTime="2026-10-30 22:06:25.123897868 +0000 UTC m=+11.693991163"
Oct 30 22:06:25.168736 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:25.168687    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v" podStartSLOduration=4.401309364 podStartE2EDuration="12.168672459s" totalImagesPullingTime="7.767363095s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:25.168552298 +0000 UTC m=+11.738645595" watchObservedRunningTime="2026-10-30 22:06:25.168672459 +0000 UTC m=+11.738765754"
Oct 30 22:06:25.873476 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:25.873401    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:25.873559 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:25.873525    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvsmv" podUID="4049e53b-bffe-40fb-beef-22acc171315a"
Oct 30 22:06:25.873618 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:25.873596    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:25.873692 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:25.873667    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcl2w" podUID="d303600a-2390-4ba9-8ea8-a58bd505a109"
Oct 30 22:06:25.873816 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:25.873800    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:25.873903 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:25.873878    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljnhp" podUID="1e444015-5346-4106-93aa-ef43eb9d0be6"
Oct 30 22:06:26.082599 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:26.082561    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-86vll" event={"ID":"801d7671-e004-4da3-898e-664eb2f1b5c3","Type":"ContainerStarted","Data":"e430c2d1cfc2a1474c8eef1c42d817d3a63f2ad24602dc743409a637ce95b88a"}
Oct 30 22:06:26.084880 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:26.084849    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p4g27" event={"ID":"ccf49970-fa36-4188-ab63-e14e440b73ec","Type":"ContainerStarted","Data":"74c2f085f19d7011fbd9edd3e5cfc390081b6a75e08cc36db5d5dde08effca19"}
Oct 30 22:06:26.087626 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:26.087604    2580 generic.go:356] "Generic (PLEG): container finished" podID="bdc6308d-a259-45db-90ea-094a8d1dd3a4" containerID="fb9521f9329cfdc5b9b4f674b8bbc068e8693e324184b9c590b97f787317ecdc" exitCode=0
Oct 30 22:06:26.087727 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:26.087682    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v69rv" event={"ID":"bdc6308d-a259-45db-90ea-094a8d1dd3a4","Type":"ContainerDied","Data":"fb9521f9329cfdc5b9b4f674b8bbc068e8693e324184b9c590b97f787317ecdc"}
Oct 30 22:06:26.123764 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:26.123672    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-86vll" podStartSLOduration=5.189745706 podStartE2EDuration="13.123658223s" totalImagesPullingTime="7.933912517s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:26.123173264 +0000 UTC m=+12.693266560" watchObservedRunningTime="2026-10-30 22:06:26.123658223 +0000 UTC m=+12.693751517"
Oct 30 22:06:26.175964 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:26.175896    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-p4g27" podStartSLOduration=3.462734023 podStartE2EDuration="13.175876857s" totalImagesPullingTime="9.713142834s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:26.175200101 +0000 UTC m=+12.745293396" watchObservedRunningTime="2026-10-30 22:06:26.175876857 +0000 UTC m=+12.745970153"
Oct 30 22:06:27.874676 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:27.874150    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:27.874676 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:27.874301    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:27.874676 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:27.874317    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvsmv" podUID="4049e53b-bffe-40fb-beef-22acc171315a"
Oct 30 22:06:27.874676 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:27.874442    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljnhp" podUID="1e444015-5346-4106-93aa-ef43eb9d0be6"
Oct 30 22:06:27.874676 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:27.874486    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:27.874676 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:27.874540    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcl2w" podUID="d303600a-2390-4ba9-8ea8-a58bd505a109"
Oct 30 22:06:28.095730 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:28.095690    2580 generic.go:356] "Generic (PLEG): container finished" podID="bdc6308d-a259-45db-90ea-094a8d1dd3a4" containerID="a680883a4e8889b474b8c6bde3333104e153f9c182b78fa821b4477df7cd6894" exitCode=0
Oct 30 22:06:28.095903 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:28.095760    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v69rv" event={"ID":"bdc6308d-a259-45db-90ea-094a8d1dd3a4","Type":"ContainerDied","Data":"a680883a4e8889b474b8c6bde3333104e153f9c182b78fa821b4477df7cd6894"}
Oct 30 22:06:29.101249 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:29.101200    2580 generic.go:356] "Generic (PLEG): container finished" podID="bdc6308d-a259-45db-90ea-094a8d1dd3a4" containerID="0fc65ce8274391c56c68dbc5d10118c05596ff8078e4775ab54ad285ed0aa7af" exitCode=0
Oct 30 22:06:29.101989 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:29.101259    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v69rv" event={"ID":"bdc6308d-a259-45db-90ea-094a8d1dd3a4","Type":"ContainerDied","Data":"0fc65ce8274391c56c68dbc5d10118c05596ff8078e4775ab54ad285ed0aa7af"}
Oct 30 22:06:29.874342 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:29.874071    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:29.874496 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:29.874143    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:29.874496 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:29.874167    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:29.874496 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:29.874466    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvsmv" podUID="4049e53b-bffe-40fb-beef-22acc171315a"
Oct 30 22:06:29.874632 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:29.874569    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljnhp" podUID="1e444015-5346-4106-93aa-ef43eb9d0be6"
Oct 30 22:06:29.874711 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:29.874682    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcl2w" podUID="d303600a-2390-4ba9-8ea8-a58bd505a109"
Oct 30 22:06:30.106838 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:30.106798    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v69rv" event={"ID":"bdc6308d-a259-45db-90ea-094a8d1dd3a4","Type":"ContainerStarted","Data":"6283688abb45f4b1075ca2b731ddd0bb435e9bd00e1e363b8703f86824c169a5"}
Oct 30 22:06:30.183431 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:30.183343    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v69rv" podStartSLOduration=7.443216885 podStartE2EDuration="17.183329093s" totalImagesPullingTime="9.740112208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:30.168545415 +0000 UTC m=+16.738638710" watchObservedRunningTime="2026-10-30 22:06:30.183329093 +0000 UTC m=+16.753422387"
Oct 30 22:06:31.111738 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:31.111705    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5" event={"ID":"47dc8b20-b520-4f4d-bd48-74b78ba300ef","Type":"ContainerStarted","Data":"09df9bce7ad291ad2181935107f53ba170d180abf3e3f6de32d13f879cfaca3b"}
Oct 30 22:06:31.112302 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:31.111788    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5" event={"ID":"47dc8b20-b520-4f4d-bd48-74b78ba300ef","Type":"ContainerStarted","Data":"bf7f543e96239519112aee5ccc165a9899f113c43be70563ba1dee84cfb4efdb"}
Oct 30 22:06:31.112302 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:31.111837    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5" event={"ID":"47dc8b20-b520-4f4d-bd48-74b78ba300ef","Type":"ContainerStarted","Data":"0f21c47c05d93439e668b91c92b6fc814e7a30f77fd796e0f710f582d2d87d2e"}
Oct 30 22:06:31.112302 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:31.111886    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5" event={"ID":"47dc8b20-b520-4f4d-bd48-74b78ba300ef","Type":"ContainerStarted","Data":"0d68c88349dec04200b602e33c63dde1957464ad2ece55856f07573a65fbaa8f"}
Oct 30 22:06:31.112302 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:31.111940    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5" event={"ID":"47dc8b20-b520-4f4d-bd48-74b78ba300ef","Type":"ContainerStarted","Data":"f8449788988bab3ff60e9c3e3a5fca6b0a83f6d153714b7f92fab6183114327c"}
Oct 30 22:06:31.112302 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:31.111979    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5" event={"ID":"47dc8b20-b520-4f4d-bd48-74b78ba300ef","Type":"ContainerStarted","Data":"e95d1cb161f6f3aca5635cbbfb017d9cabeccbb5dac6f2e7074edd0dc26a2ea8"}
Oct 30 22:06:31.641517 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:31.641334    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e444015-5346-4106-93aa-ef43eb9d0be6-original-pull-secret\") pod \"global-pull-secret-syncer-ljnhp\" (UID: \"1e444015-5346-4106-93aa-ef43eb9d0be6\") " pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:31.641665 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:31.641536    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs\") pod \"network-metrics-daemon-fvsmv\" (UID: \"4049e53b-bffe-40fb-beef-22acc171315a\") " pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:31.641665 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:31.641481    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:31.641665 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:31.641622    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:31.641665 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:31.641635    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e444015-5346-4106-93aa-ef43eb9d0be6-original-pull-secret podName:1e444015-5346-4106-93aa-ef43eb9d0be6 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:47.641619265 +0000 UTC m=+34.211712552 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e444015-5346-4106-93aa-ef43eb9d0be6-original-pull-secret") pod "global-pull-secret-syncer-ljnhp" (UID: "1e444015-5346-4106-93aa-ef43eb9d0be6") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:31.641818 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:31.641672    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs podName:4049e53b-bffe-40fb-beef-22acc171315a nodeName:}" failed. No retries permitted until 2026-10-30 22:06:47.641662721 +0000 UTC m=+34.211755994 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs") pod "network-metrics-daemon-fvsmv" (UID: "4049e53b-bffe-40fb-beef-22acc171315a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:31.742488 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:31.742456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wthz4\" (UniqueName: \"kubernetes.io/projected/d303600a-2390-4ba9-8ea8-a58bd505a109-kube-api-access-wthz4\") pod \"network-check-target-zcl2w\" (UID: \"d303600a-2390-4ba9-8ea8-a58bd505a109\") " pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:31.742643 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:31.742599    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:06:31.742643 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:31.742616    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:06:31.742643 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:31.742625    2580 projected.go:196] Error preparing data for projected volume kube-api-access-wthz4 for pod openshift-network-diagnostics/network-check-target-zcl2w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:31.742733 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:31.742668    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d303600a-2390-4ba9-8ea8-a58bd505a109-kube-api-access-wthz4 podName:d303600a-2390-4ba9-8ea8-a58bd505a109 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:47.74265657 +0000 UTC m=+34.312749843 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wthz4" (UniqueName: "kubernetes.io/projected/d303600a-2390-4ba9-8ea8-a58bd505a109-kube-api-access-wthz4") pod "network-check-target-zcl2w" (UID: "d303600a-2390-4ba9-8ea8-a58bd505a109") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:31.874436 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:31.874398    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:31.874436 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:31.874425    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:31.874649 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:31.874508    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvsmv" podUID="4049e53b-bffe-40fb-beef-22acc171315a"
Oct 30 22:06:31.874649 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:31.874565    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:31.874793 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:31.874644    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcl2w" podUID="d303600a-2390-4ba9-8ea8-a58bd505a109"
Oct 30 22:06:31.874793 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:31.874675    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljnhp" podUID="1e444015-5346-4106-93aa-ef43eb9d0be6"
Oct 30 22:06:33.120130 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:33.120098    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5" event={"ID":"47dc8b20-b520-4f4d-bd48-74b78ba300ef","Type":"ContainerStarted","Data":"6c213269079ad757223ed6a0bf4f561f0954916569bcab6d182a6c0ea6fc1cbd"}
Oct 30 22:06:33.874800 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:33.874772    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:33.874965 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:33.874868    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljnhp" podUID="1e444015-5346-4106-93aa-ef43eb9d0be6"
Oct 30 22:06:33.874965 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:33.874938    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:33.875033 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:33.875017    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcl2w" podUID="d303600a-2390-4ba9-8ea8-a58bd505a109"
Oct 30 22:06:33.875068 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:33.875032    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:33.875154 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:33.875128    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvsmv" podUID="4049e53b-bffe-40fb-beef-22acc171315a"
Oct 30 22:06:35.873619 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:35.873589    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:35.874460 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:35.873622    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:35.874460 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:35.873681    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcl2w" podUID="d303600a-2390-4ba9-8ea8-a58bd505a109"
Oct 30 22:06:35.874460 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:35.873714    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:35.874460 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:35.873793    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljnhp" podUID="1e444015-5346-4106-93aa-ef43eb9d0be6"
Oct 30 22:06:35.874460 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:35.873883    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvsmv" podUID="4049e53b-bffe-40fb-beef-22acc171315a"
Oct 30 22:06:36.129367 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:36.129282    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5" event={"ID":"47dc8b20-b520-4f4d-bd48-74b78ba300ef","Type":"ContainerStarted","Data":"5dba64af5ab6d3086f0529923d0f070803eb1c50eb9e3297b5d5e654feccd7df"}
Oct 30 22:06:36.129547 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:36.129534    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:36.145605 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:36.145581    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:36.223856 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:36.223805    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5" podStartSLOduration=8.820958151 podStartE2EDuration="23.223792164s" totalImagesPullingTime="14.402834013s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:36.223423325 +0000 UTC m=+22.793516619" watchObservedRunningTime="2026-10-30 22:06:36.223792164 +0000 UTC m=+22.793885458"
Oct 30 22:06:37.132600 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:37.132574    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:37.133086 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:37.132693    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:37.149388 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:37.149361    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:37.181490 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:37.181448    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ljnhp"]
Oct 30 22:06:37.181645 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:37.181572    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:37.181698 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:37.181659    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljnhp" podUID="1e444015-5346-4106-93aa-ef43eb9d0be6"
Oct 30 22:06:37.195361 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:37.195324    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fvsmv"]
Oct 30 22:06:37.195504 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:37.195449    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:37.195588 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:37.195568    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvsmv" podUID="4049e53b-bffe-40fb-beef-22acc171315a"
Oct 30 22:06:37.195902 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:37.195884    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zcl2w"]
Oct 30 22:06:37.195981 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:37.195966    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:37.196070 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:37.196053    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcl2w" podUID="d303600a-2390-4ba9-8ea8-a58bd505a109"
Oct 30 22:06:38.873947 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:38.873755    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:38.874424 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:38.873839    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:38.874424 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:38.874015    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcl2w" podUID="d303600a-2390-4ba9-8ea8-a58bd505a109"
Oct 30 22:06:38.874424 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:38.874134    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvsmv" podUID="4049e53b-bffe-40fb-beef-22acc171315a"
Oct 30 22:06:38.874424 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:38.873847    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:38.874424 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:38.874240    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljnhp" podUID="1e444015-5346-4106-93aa-ef43eb9d0be6"
Oct 30 22:06:40.873534 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:40.873498    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:40.873964 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:40.873601    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:40.873964 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:40.873619    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcl2w" podUID="d303600a-2390-4ba9-8ea8-a58bd505a109"
Oct 30 22:06:40.873964 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:40.873630    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:40.873964 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:40.873719    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljnhp" podUID="1e444015-5346-4106-93aa-ef43eb9d0be6"
Oct 30 22:06:40.873964 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:40.873788    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvsmv" podUID="4049e53b-bffe-40fb-beef-22acc171315a"
Oct 30 22:06:42.874094 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:42.874050    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:42.874094 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:42.874079    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:42.874656 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:42.874181    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:42.874656 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:42.874179    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljnhp" podUID="1e444015-5346-4106-93aa-ef43eb9d0be6"
Oct 30 22:06:42.874656 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:42.874324    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvsmv" podUID="4049e53b-bffe-40fb-beef-22acc171315a"
Oct 30 22:06:42.874656 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:42.874411    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcl2w" podUID="d303600a-2390-4ba9-8ea8-a58bd505a109"
Oct 30 22:06:43.281036 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.281006    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeReady"
Oct 30 22:06:43.281190 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.281153    2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 22:06:43.327196 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.327162    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dvcrc"]
Oct 30 22:06:43.330266 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.330240    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-k8lw5"]
Oct 30 22:06:43.330371 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.330315    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-dvcrc"
Oct 30 22:06:43.332711 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.332694    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-74sl2"]
Oct 30 22:06:43.332829 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.332818    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-k8lw5"
Oct 30 22:06:43.335310 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.335289    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-74sl2"
Oct 30 22:06:43.336994 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.336974    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jh5sc"]
Oct 30 22:06:43.340739 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.340721    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-fb457cdb7-m99g8"]
Oct 30 22:06:43.340880 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.340840    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jh5sc"
Oct 30 22:06:43.344269 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.344251    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-t2gq6"]
Oct 30 22:06:43.344388 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.344370    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:43.347329 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.347311    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"]
Oct 30 22:06:43.347438 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.347424    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:43.350083 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.350064    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-vwshh"]
Oct 30 22:06:43.350213 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.350198    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"
Oct 30 22:06:43.352842 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.352735    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-54b978dcc8-sch2b"]
Oct 30 22:06:43.352914 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.352849    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:43.356351 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.355681    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-48hjx"]
Oct 30 22:06:43.356351 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.355860    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 22:06:43.356351 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.356323    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 22:06:43.356507 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.356394    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:43.359365 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.359345    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"]
Oct 30 22:06:43.359523 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.359451    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-48hjx"
Oct 30 22:06:43.365670 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.365648    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-jd958"]
Oct 30 22:06:43.366058 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.366033    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"
Oct 30 22:06:43.369516 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.369483    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 22:06:43.369516 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.369508    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 22:06:43.369753 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.369612    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 22:06:43.369753 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.369667    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 22:06:43.370427 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.370089    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z6gtt\""
Oct 30 22:06:43.373065 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.372113    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-qbz82"]
Oct 30 22:06:43.373065 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.372795    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jd958"
Oct 30 22:06:43.377467 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.377447    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-n42rm"]
Oct 30 22:06:43.378540 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.378518    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:06:43.381552 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.381439    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-9xdrk"]
Oct 30 22:06:43.381645 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.381534    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-n42rm"
Oct 30 22:06:43.385773 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.385755    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b9d7c7dff-82x2r"]
Oct 30 22:06:43.385887 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.385869    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9xdrk"
Oct 30 22:06:43.389456 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.389436    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-p89nn"]
Oct 30 22:06:43.389808 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.389782    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:43.392902 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.392885    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dvcrc"]
Oct 30 22:06:43.394666 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.394652    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-74sl2"]
Oct 30 22:06:43.395122 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.395106    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-k8lw5"]
Oct 30 22:06:43.395216 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.395129    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jh5sc"]
Oct 30 22:06:43.395216 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.395143    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-fb457cdb7-m99g8"]
Oct 30 22:06:43.395216 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.393099    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p89nn"
Oct 30 22:06:43.395216 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.395161    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"]
Oct 30 22:06:43.395216 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.395177    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-vwshh"]
Oct 30 22:06:43.395216 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.395201    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-jd958"]
Oct 30 22:06:43.395216 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.395236    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"]
Oct 30 22:06:43.395576 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.395263    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54b978dcc8-sch2b"]
Oct 30 22:06:43.395576 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.395283    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p89nn"]
Oct 30 22:06:43.395576 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.395303    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t2gq6"]
Oct 30 22:06:43.395576 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.395326    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b9d7c7dff-82x2r"]
Oct 30 22:06:43.395576 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.395350    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-48hjx"]
Oct 30 22:06:43.395576 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.395372    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-n42rm"]
Oct 30 22:06:43.395576 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.395391    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-9xdrk"]
Oct 30 22:06:43.397065 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.397047    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-qbz82"]
Oct 30 22:06:43.422638 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.422615    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/965e7b7f-9044-4c69-be4d-06ef9a1044e2-service-ca-bundle\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:43.422749 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.422647    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e297483a-0b86-4481-9597-0c89e9a6a7ac-image-registry-private-configuration\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:43.422749 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.422732    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:43.422862 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.422764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6cjm\" (UniqueName: \"kubernetes.io/projected/200052bd-2f0a-46d2-8b9b-6bd250b17b05-kube-api-access-h6cjm\") pod \"cluster-monitoring-operator-884576c57-qc4bw\" (UID: \"200052bd-2f0a-46d2-8b9b-6bd250b17b05\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"
Oct 30 22:06:43.422862 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.422824    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e297483a-0b86-4481-9597-0c89e9a6a7ac-ca-trust-extracted\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:43.422955 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.422890    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-bound-sa-token\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:43.422955 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.422909    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcmwp\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-kube-api-access-vcmwp\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:43.423064 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.422959    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrm4r\" (UniqueName: \"kubernetes.io/projected/88fe4dfd-dcda-444d-871b-24ca3fb10d77-kube-api-access-xrm4r\") pod \"volume-data-source-validator-58c9f5f46d-jh5sc\" (UID: \"88fe4dfd-dcda-444d-871b-24ca3fb10d77\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jh5sc"
Oct 30 22:06:43.423064 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.422997    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/965e7b7f-9044-4c69-be4d-06ef9a1044e2-serving-cert\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:43.423064 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.423035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7wm8\" (UniqueName: \"kubernetes.io/projected/b8ee94be-1472-42e7-bf6e-d13e58671eeb-kube-api-access-h7wm8\") pod \"kas-connection-checker-685895c7c9-dvcrc\" (UID: \"b8ee94be-1472-42e7-bf6e-d13e58671eeb\") " pod="kube-system/kas-connection-checker-685895c7c9-dvcrc"
Oct 30 22:06:43.423236 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.423099    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv2fd\" (UniqueName: \"kubernetes.io/projected/c31a736c-cd3e-42be-b783-fc25436b5302-kube-api-access-vv2fd\") pod \"dns-default-t2gq6\" (UID: \"c31a736c-cd3e-42be-b783-fc25436b5302\") " pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:43.423236 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.423116    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:43.423236 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.423133    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdqmz\" (UniqueName: \"kubernetes.io/projected/3f403b34-f155-49e6-b327-8ad89d00bbc3-kube-api-access-qdqmz\") pod \"kas-connection-checker-685895c7c9-k8lw5\" (UID: \"3f403b34-f155-49e6-b327-8ad89d00bbc3\") " pod="kube-system/kas-connection-checker-685895c7c9-k8lw5"
Oct 30 22:06:43.423236 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.423150    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/200052bd-2f0a-46d2-8b9b-6bd250b17b05-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qc4bw\" (UID: \"200052bd-2f0a-46d2-8b9b-6bd250b17b05\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"
Oct 30 22:06:43.423236 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.423166    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/965e7b7f-9044-4c69-be4d-06ef9a1044e2-tmp\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:43.423480 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.423238    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr2ww\" (UniqueName: \"kubernetes.io/projected/605c1b60-1fa5-4257-a588-4b2e0a78bcca-kube-api-access-dr2ww\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:43.423480 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.423274    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/965e7b7f-9044-4c69-be4d-06ef9a1044e2-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:43.423480 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.423303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qc4bw\" (UID: \"200052bd-2f0a-46d2-8b9b-6bd250b17b05\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"
Oct 30 22:06:43.423480 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.423392    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-default-certificate\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:43.423480 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.423416    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-certificates\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:43.423480 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.423433    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c31a736c-cd3e-42be-b783-fc25436b5302-config-volume\") pod \"dns-default-t2gq6\" (UID: \"c31a736c-cd3e-42be-b783-fc25436b5302\") " pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:43.423480 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.423453    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c31a736c-cd3e-42be-b783-fc25436b5302-metrics-tls\") pod \"dns-default-t2gq6\" (UID: \"c31a736c-cd3e-42be-b783-fc25436b5302\") " pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:43.423480 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.423474    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c31a736c-cd3e-42be-b783-fc25436b5302-tmp-dir\") pod \"dns-default-t2gq6\" (UID: \"c31a736c-cd3e-42be-b783-fc25436b5302\") " pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:43.423842 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.423497    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b66td\" (UniqueName: \"kubernetes.io/projected/00b74f51-6c71-45f2-97af-6aea20b9eeaf-kube-api-access-b66td\") pod \"kas-connection-checker-685895c7c9-74sl2\" (UID: \"00b74f51-6c71-45f2-97af-6aea20b9eeaf\") " pod="kube-system/kas-connection-checker-685895c7c9-74sl2"
Oct 30 22:06:43.423842 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.423522    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605c1b60-1fa5-4257-a588-4b2e0a78bcca-service-ca-bundle\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:43.423842 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.423562    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e297483a-0b86-4481-9597-0c89e9a6a7ac-trusted-ca\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:43.423842 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.423613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e297483a-0b86-4481-9597-0c89e9a6a7ac-installation-pull-secrets\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:43.423842 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.423665    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-stats-auth\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:43.423842 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.423681    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/965e7b7f-9044-4c69-be4d-06ef9a1044e2-snapshots\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:43.423842 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.423695    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qqhv\" (UniqueName: \"kubernetes.io/projected/965e7b7f-9044-4c69-be4d-06ef9a1044e2-kube-api-access-7qqhv\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:43.451409 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.451387    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 22:06:43.451518 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.451467    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 22:06:43.451518 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.451498    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 22:06:43.451623 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.451512    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:06:43.451623 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.451574    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 22:06:43.451848 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.451833    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 22:06:43.451848 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.451843    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 22:06:43.451972 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.451836    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 22:06:43.452288 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.452268    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:06:43.452369 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.452325    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:06:43.452438 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.452393    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 22:06:43.452438 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.452422    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 22:06:43.452600 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.452580    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 22:06:43.452734 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.452679    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 22:06:43.452811 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.452758    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:06:43.452990 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.452965    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 22:06:43.453118 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.453101    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 22:06:43.453404 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.453390    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 22:06:43.453460 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.453428    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 22:06:43.453460 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.453451    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 22:06:43.453525 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.453486    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 22:06:43.453562 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.453546    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:06:43.454297 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.454015    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 22:06:43.454297 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.454033    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 22:06:43.454297 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.454070    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 22:06:43.454297 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.454071    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 22:06:43.454297 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.454180    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 22:06:43.454297 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.454253    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 22:06:43.466210 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.466190    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 22:06:43.481196 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.481163    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 22:06:43.481333 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.481258    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 22:06:43.481333 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.481296    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 22:06:43.481333 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.481324    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 22:06:43.481477 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.481374    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-49l9d\""
Oct 30 22:06:43.481477 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.481396    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 22:06:43.481477 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.481407    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 22:06:43.481477 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.481432    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 22:06:43.481477 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.481456    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-htsvp\""
Oct 30 22:06:43.481691 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.481486    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 22:06:43.481691 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.481501    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 22:06:43.481691 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.481519    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 22:06:43.481691 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.481565    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-rz55v\""
Oct 30 22:06:43.481691 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.481579    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 22:06:43.481691 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.481687    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 22:06:43.481891 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.481708    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 22:06:43.481891 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.481761    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 22:06:43.524885 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.524854    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv2fd\" (UniqueName: \"kubernetes.io/projected/c31a736c-cd3e-42be-b783-fc25436b5302-kube-api-access-vv2fd\") pod \"dns-default-t2gq6\" (UID: \"c31a736c-cd3e-42be-b783-fc25436b5302\") " pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:43.524885 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.524884    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:43.525081 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.524905    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdqmz\" (UniqueName: \"kubernetes.io/projected/3f403b34-f155-49e6-b327-8ad89d00bbc3-kube-api-access-qdqmz\") pod \"kas-connection-checker-685895c7c9-k8lw5\" (UID: \"3f403b34-f155-49e6-b327-8ad89d00bbc3\") " pod="kube-system/kas-connection-checker-685895c7c9-k8lw5"
Oct 30 22:06:43.525081 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.524923    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/200052bd-2f0a-46d2-8b9b-6bd250b17b05-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qc4bw\" (UID: \"200052bd-2f0a-46d2-8b9b-6bd250b17b05\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"
Oct 30 22:06:43.525081 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.524946    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/965e7b7f-9044-4c69-be4d-06ef9a1044e2-tmp\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:43.525081 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.524974    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bptmv\" (UniqueName: \"kubernetes.io/projected/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-kube-api-access-bptmv\") pod \"cluster-samples-operator-6ddb6c8b94-vqtw4\" (UID: \"fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"
Oct 30 22:06:43.525081 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr2ww\" (UniqueName: \"kubernetes.io/projected/605c1b60-1fa5-4257-a588-4b2e0a78bcca-kube-api-access-dr2ww\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:43.525081 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:43.525016    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:06:43.525081 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/965e7b7f-9044-4c69-be4d-06ef9a1044e2-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:43.525081 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:43.525037    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54b978dcc8-sch2b: secret "image-registry-tls" not found
Oct 30 22:06:43.525081 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525049    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:43.525081 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525073    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5b84639d-6b5d-4795-a356-8755ea234585-config\") pod \"service-ca-operator-544d749c8c-jd958\" (UID: \"5b84639d-6b5d-4795-a356-8755ea234585\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jd958"
Oct 30 22:06:43.525591 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:43.525104    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls podName:e297483a-0b86-4481-9597-0c89e9a6a7ac nodeName:}" failed. No retries permitted until 2026-10-30 22:06:44.025086107 +0000 UTC m=+30.595179391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls") pod "image-registry-54b978dcc8-sch2b" (UID: "e297483a-0b86-4481-9597-0c89e9a6a7ac") : secret "image-registry-tls" not found
Oct 30 22:06:43.525591 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525128    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qc4bw\" (UID: \"200052bd-2f0a-46d2-8b9b-6bd250b17b05\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"
Oct 30 22:06:43.525591 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525167    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vqtw4\" (UID: \"fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"
Oct 30 22:06:43.525591 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:43.525268    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:06:43.525591 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525293    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0c1f2fe-ca1b-4556-8df1-134c67cee4f8-cert\") pod \"ingress-canary-p89nn\" (UID: \"e0c1f2fe-ca1b-4556-8df1-134c67cee4f8\") " pod="openshift-ingress-canary/ingress-canary-p89nn"
Oct 30 22:06:43.525591 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:43.525312    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls podName:200052bd-2f0a-46d2-8b9b-6bd250b17b05 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:44.025299871 +0000 UTC m=+30.595393150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qc4bw" (UID: "200052bd-2f0a-46d2-8b9b-6bd250b17b05") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:06:43.525591 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525346    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-default-certificate\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:43.525591 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525368    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-certificates\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:43.525591 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea4b0705-2096-41da-b78b-b62590f236e8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-n42rm\" (UID: \"ea4b0705-2096-41da-b78b-b62590f236e8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-n42rm"
Oct 30 22:06:43.525591 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525414    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h824x\" (UniqueName: \"kubernetes.io/projected/24c987a0-1cbf-4264-ba0e-586b738d08da-kube-api-access-h824x\") pod \"network-check-source-6f78d59f99-9xdrk\" (UID: \"24c987a0-1cbf-4264-ba0e-586b738d08da\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9xdrk"
Oct 30 22:06:43.525591 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525426    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/965e7b7f-9044-4c69-be4d-06ef9a1044e2-tmp\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:43.525591 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525434    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c31a736c-cd3e-42be-b783-fc25436b5302-config-volume\") pod \"dns-default-t2gq6\" (UID: \"c31a736c-cd3e-42be-b783-fc25436b5302\") " pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:43.525591 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525482    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c31a736c-cd3e-42be-b783-fc25436b5302-metrics-tls\") pod \"dns-default-t2gq6\" (UID: \"c31a736c-cd3e-42be-b783-fc25436b5302\") " pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:43.525591 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525514    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c31a736c-cd3e-42be-b783-fc25436b5302-tmp-dir\") pod \"dns-default-t2gq6\" (UID: \"c31a736c-cd3e-42be-b783-fc25436b5302\") " pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:43.525591 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525546    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr8dq\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-kube-api-access-vr8dq\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:43.526310 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ea9c808c-62b4-4dc7-9761-bf6ee3700e34-config\") pod \"console-operator-7cd56b8dbc-qbz82\" (UID: \"ea9c808c-62b4-4dc7-9761-bf6ee3700e34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:06:43.526310 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525600    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea9c808c-62b4-4dc7-9761-bf6ee3700e34-serving-cert\") pod \"console-operator-7cd56b8dbc-qbz82\" (UID: \"ea9c808c-62b4-4dc7-9761-bf6ee3700e34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:06:43.526310 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525629    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b66td\" (UniqueName: \"kubernetes.io/projected/00b74f51-6c71-45f2-97af-6aea20b9eeaf-kube-api-access-b66td\") pod \"kas-connection-checker-685895c7c9-74sl2\" (UID: \"00b74f51-6c71-45f2-97af-6aea20b9eeaf\") " pod="kube-system/kas-connection-checker-685895c7c9-74sl2"
Oct 30 22:06:43.526310 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525655    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605c1b60-1fa5-4257-a588-4b2e0a78bcca-service-ca-bundle\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:43.526310 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525680    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmhwk\" (UniqueName: \"kubernetes.io/projected/f14d2c99-6d70-44bf-9046-7cc1b84f1e7e-kube-api-access-tmhwk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-48hjx\" (UID: \"f14d2c99-6d70-44bf-9046-7cc1b84f1e7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-48hjx"
Oct 30 22:06:43.526310 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525707    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e297483a-0b86-4481-9597-0c89e9a6a7ac-trusted-ca\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:43.526310 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e297483a-0b86-4481-9597-0c89e9a6a7ac-installation-pull-secrets\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:43.526310 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525763    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b6651a05-7ba7-484f-8490-f22665bcb84f-image-registry-private-configuration\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:43.526310 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525802    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ea4b0705-2096-41da-b78b-b62590f236e8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-n42rm\" (UID: \"ea4b0705-2096-41da-b78b-b62590f236e8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-n42rm"
Oct 30 22:06:43.526310 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525826    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f14d2c99-6d70-44bf-9046-7cc1b84f1e7e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-48hjx\" (UID: \"f14d2c99-6d70-44bf-9046-7cc1b84f1e7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-48hjx"
Oct 30 22:06:43.526310 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525850    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f14d2c99-6d70-44bf-9046-7cc1b84f1e7e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-48hjx\" (UID: \"f14d2c99-6d70-44bf-9046-7cc1b84f1e7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-48hjx"
Oct 30 22:06:43.526310 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-stats-auth\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:43.526310 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525907    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/965e7b7f-9044-4c69-be4d-06ef9a1044e2-snapshots\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:43.526310 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525932    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qqhv\" (UniqueName: \"kubernetes.io/projected/965e7b7f-9044-4c69-be4d-06ef9a1044e2-kube-api-access-7qqhv\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:43.526310 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525936    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c31a736c-cd3e-42be-b783-fc25436b5302-config-volume\") pod \"dns-default-t2gq6\" (UID: \"c31a736c-cd3e-42be-b783-fc25436b5302\") " pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:43.527014 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525955    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-bound-sa-token\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:43.527014 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.526000    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-certificates\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:43.527014 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.525987    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/200052bd-2f0a-46d2-8b9b-6bd250b17b05-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qc4bw\" (UID: \"200052bd-2f0a-46d2-8b9b-6bd250b17b05\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"
Oct 30 22:06:43.527014 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.526036    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/965e7b7f-9044-4c69-be4d-06ef9a1044e2-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:43.527014 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.526059    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-certificates\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:43.527014 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:43.526207    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:06:43.527014 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:43.526301    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c31a736c-cd3e-42be-b783-fc25436b5302-metrics-tls podName:c31a736c-cd3e-42be-b783-fc25436b5302 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:44.026286657 +0000 UTC m=+30.596379951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c31a736c-cd3e-42be-b783-fc25436b5302-metrics-tls") pod "dns-default-t2gq6" (UID: "c31a736c-cd3e-42be-b783-fc25436b5302") : secret "dns-default-metrics-tls" not found
Oct 30 22:06:43.527014 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.526388    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7gv7\" (UniqueName: \"kubernetes.io/projected/ea9c808c-62b4-4dc7-9761-bf6ee3700e34-kube-api-access-j7gv7\") pod \"console-operator-7cd56b8dbc-qbz82\" (UID: \"ea9c808c-62b4-4dc7-9761-bf6ee3700e34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:06:43.527014 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.526429    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b6651a05-7ba7-484f-8490-f22665bcb84f-installation-pull-secrets\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:43.527014 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.526453    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vqtw4\" (UID: \"fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"
Oct 30 22:06:43.527014 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.526494    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/965e7b7f-9044-4c69-be4d-06ef9a1044e2-service-ca-bundle\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:43.527014 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.526496    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c31a736c-cd3e-42be-b783-fc25436b5302-tmp-dir\") pod \"dns-default-t2gq6\" (UID: \"c31a736c-cd3e-42be-b783-fc25436b5302\") " pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:43.527014 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.526517    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ea9c808c-62b4-4dc7-9761-bf6ee3700e34-trusted-ca\") pod \"console-operator-7cd56b8dbc-qbz82\" (UID: \"ea9c808c-62b4-4dc7-9761-bf6ee3700e34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:06:43.527014 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.526548    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/965e7b7f-9044-4c69-be4d-06ef9a1044e2-snapshots\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:43.527014 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.526557    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e297483a-0b86-4481-9597-0c89e9a6a7ac-image-registry-private-configuration\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:43.527014 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.526698    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:43.527738 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.526736    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6cjm\" (UniqueName: \"kubernetes.io/projected/200052bd-2f0a-46d2-8b9b-6bd250b17b05-kube-api-access-h6cjm\") pod \"cluster-monitoring-operator-884576c57-qc4bw\" (UID: \"200052bd-2f0a-46d2-8b9b-6bd250b17b05\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"
Oct 30 22:06:43.527738 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.526767    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk8vc\" (UniqueName: \"kubernetes.io/projected/e0c1f2fe-ca1b-4556-8df1-134c67cee4f8-kube-api-access-lk8vc\") pod \"ingress-canary-p89nn\" (UID: \"e0c1f2fe-ca1b-4556-8df1-134c67cee4f8\") " pod="openshift-ingress-canary/ingress-canary-p89nn"
Oct 30 22:06:43.527738 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:43.526866    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:06:43.527738 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:43.526903    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs podName:605c1b60-1fa5-4257-a588-4b2e0a78bcca nodeName:}" failed. No retries permitted until 2026-10-30 22:06:44.026891347 +0000 UTC m=+30.596984637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs") pod "router-default-fb457cdb7-m99g8" (UID: "605c1b60-1fa5-4257-a588-4b2e0a78bcca") : secret "router-metrics-certs-default" not found
Oct 30 22:06:43.527738 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.526944    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e297483a-0b86-4481-9597-0c89e9a6a7ac-trusted-ca\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:43.527738 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.526968    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/965e7b7f-9044-4c69-be4d-06ef9a1044e2-service-ca-bundle\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:43.527738 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.527131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e297483a-0b86-4481-9597-0c89e9a6a7ac-ca-trust-extracted\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:43.527738 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.527207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-bound-sa-token\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:43.527738 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:43.527255    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/605c1b60-1fa5-4257-a588-4b2e0a78bcca-service-ca-bundle podName:605c1b60-1fa5-4257-a588-4b2e0a78bcca nodeName:}" failed. No retries permitted until 2026-10-30 22:06:44.027241794 +0000 UTC m=+30.597335078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/605c1b60-1fa5-4257-a588-4b2e0a78bcca-service-ca-bundle") pod "router-default-fb457cdb7-m99g8" (UID: "605c1b60-1fa5-4257-a588-4b2e0a78bcca") : configmap references non-existent config key: service-ca.crt
Oct 30 22:06:43.527738 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.527291    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcmwp\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-kube-api-access-vcmwp\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:43.527738 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.527612    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrm4r\" (UniqueName: \"kubernetes.io/projected/88fe4dfd-dcda-444d-871b-24ca3fb10d77-kube-api-access-xrm4r\") pod \"volume-data-source-validator-58c9f5f46d-jh5sc\" (UID: \"88fe4dfd-dcda-444d-871b-24ca3fb10d77\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jh5sc"
Oct 30 22:06:43.527738 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.527623    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e297483a-0b86-4481-9597-0c89e9a6a7ac-ca-trust-extracted\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:43.527738 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.527681    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b6651a05-7ba7-484f-8490-f22665bcb84f-ca-trust-extracted\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:43.528421 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.527788    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6651a05-7ba7-484f-8490-f22665bcb84f-trusted-ca\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:43.528421 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.527823    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5b84639d-6b5d-4795-a356-8755ea234585-serving-cert\") pod \"service-ca-operator-544d749c8c-jd958\" (UID: \"5b84639d-6b5d-4795-a356-8755ea234585\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jd958"
Oct 30 22:06:43.528421 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.527923    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/965e7b7f-9044-4c69-be4d-06ef9a1044e2-serving-cert\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:43.528421 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.528074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7wm8\" (UniqueName: \"kubernetes.io/projected/b8ee94be-1472-42e7-bf6e-d13e58671eeb-kube-api-access-h7wm8\") pod \"kas-connection-checker-685895c7c9-dvcrc\" (UID: \"b8ee94be-1472-42e7-bf6e-d13e58671eeb\") " pod="kube-system/kas-connection-checker-685895c7c9-dvcrc"
Oct 30 22:06:43.528421 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.528104    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92nhs\" (UniqueName: \"kubernetes.io/projected/5b84639d-6b5d-4795-a356-8755ea234585-kube-api-access-92nhs\") pod \"service-ca-operator-544d749c8c-jd958\" (UID: \"5b84639d-6b5d-4795-a356-8755ea234585\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jd958"
Oct 30 22:06:43.530039 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.530014    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e297483a-0b86-4481-9597-0c89e9a6a7ac-image-registry-private-configuration\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:43.530133 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.530038    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e297483a-0b86-4481-9597-0c89e9a6a7ac-installation-pull-secrets\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:43.530133 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.530042    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-default-certificate\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:43.530133 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.530038    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-stats-auth\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:43.530329 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.530312    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/965e7b7f-9044-4c69-be4d-06ef9a1044e2-serving-cert\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:43.535388 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.535363    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:43.546776 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.546745    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv2fd\" (UniqueName: \"kubernetes.io/projected/c31a736c-cd3e-42be-b783-fc25436b5302-kube-api-access-vv2fd\") pod \"dns-default-t2gq6\" (UID: \"c31a736c-cd3e-42be-b783-fc25436b5302\") " pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:43.547425 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.547399    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdqmz\" (UniqueName: \"kubernetes.io/projected/3f403b34-f155-49e6-b327-8ad89d00bbc3-kube-api-access-qdqmz\") pod \"kas-connection-checker-685895c7c9-k8lw5\" (UID: \"3f403b34-f155-49e6-b327-8ad89d00bbc3\") " pod="kube-system/kas-connection-checker-685895c7c9-k8lw5"
Oct 30 22:06:43.547666 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.547639    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr2ww\" (UniqueName: \"kubernetes.io/projected/605c1b60-1fa5-4257-a588-4b2e0a78bcca-kube-api-access-dr2ww\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:43.548375 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.548332    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6cjm\" (UniqueName: \"kubernetes.io/projected/200052bd-2f0a-46d2-8b9b-6bd250b17b05-kube-api-access-h6cjm\") pod \"cluster-monitoring-operator-884576c57-qc4bw\" (UID: \"200052bd-2f0a-46d2-8b9b-6bd250b17b05\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"
Oct 30 22:06:43.548752 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.548722    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qqhv\" (UniqueName: \"kubernetes.io/projected/965e7b7f-9044-4c69-be4d-06ef9a1044e2-kube-api-access-7qqhv\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:43.548831 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.548763    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b66td\" (UniqueName: \"kubernetes.io/projected/00b74f51-6c71-45f2-97af-6aea20b9eeaf-kube-api-access-b66td\") pod \"kas-connection-checker-685895c7c9-74sl2\" (UID: \"00b74f51-6c71-45f2-97af-6aea20b9eeaf\") " pod="kube-system/kas-connection-checker-685895c7c9-74sl2"
Oct 30 22:06:43.549374 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.549355    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrm4r\" (UniqueName: \"kubernetes.io/projected/88fe4dfd-dcda-444d-871b-24ca3fb10d77-kube-api-access-xrm4r\") pod \"volume-data-source-validator-58c9f5f46d-jh5sc\" (UID: \"88fe4dfd-dcda-444d-871b-24ca3fb10d77\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jh5sc"
Oct 30 22:06:43.549802 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.549777    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7wm8\" (UniqueName: \"kubernetes.io/projected/b8ee94be-1472-42e7-bf6e-d13e58671eeb-kube-api-access-h7wm8\") pod \"kas-connection-checker-685895c7c9-dvcrc\" (UID: \"b8ee94be-1472-42e7-bf6e-d13e58671eeb\") " pod="kube-system/kas-connection-checker-685895c7c9-dvcrc"
Oct 30 22:06:43.549943 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.549921    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcmwp\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-kube-api-access-vcmwp\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:43.550507 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.550492    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-bound-sa-token\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:43.628567 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.628533    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:43.628567 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.628569    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5b84639d-6b5d-4795-a356-8755ea234585-config\") pod \"service-ca-operator-544d749c8c-jd958\" (UID: \"5b84639d-6b5d-4795-a356-8755ea234585\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jd958"
Oct 30 22:06:43.628760 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.628597    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vqtw4\" (UID: \"fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"
Oct 30 22:06:43.628760 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.628617    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0c1f2fe-ca1b-4556-8df1-134c67cee4f8-cert\") pod \"ingress-canary-p89nn\" (UID: \"e0c1f2fe-ca1b-4556-8df1-134c67cee4f8\") " pod="openshift-ingress-canary/ingress-canary-p89nn"
Oct 30 22:06:43.628760 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:43.628691    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:06:43.628760 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:43.628708    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b9d7c7dff-82x2r: secret "image-registry-tls" not found
Oct 30 22:06:43.628760 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:43.628758    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls podName:b6651a05-7ba7-484f-8490-f22665bcb84f nodeName:}" failed. No retries permitted until 2026-10-30 22:06:44.128743122 +0000 UTC m=+30.698836422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls") pod "image-registry-6b9d7c7dff-82x2r" (UID: "b6651a05-7ba7-484f-8490-f22665bcb84f") : secret "image-registry-tls" not found
Oct 30 22:06:43.628953 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.628827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea4b0705-2096-41da-b78b-b62590f236e8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-n42rm\" (UID: \"ea4b0705-2096-41da-b78b-b62590f236e8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-n42rm"
Oct 30 22:06:43.628953 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.628862    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h824x\" (UniqueName: \"kubernetes.io/projected/24c987a0-1cbf-4264-ba0e-586b738d08da-kube-api-access-h824x\") pod \"network-check-source-6f78d59f99-9xdrk\" (UID: \"24c987a0-1cbf-4264-ba0e-586b738d08da\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9xdrk"
Oct 30 22:06:43.628953 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.628905    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr8dq\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-kube-api-access-vr8dq\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:43.628953 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.628932    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ea9c808c-62b4-4dc7-9761-bf6ee3700e34-config\") pod \"console-operator-7cd56b8dbc-qbz82\" (UID: \"ea9c808c-62b4-4dc7-9761-bf6ee3700e34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:06:43.628953 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.628952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea9c808c-62b4-4dc7-9761-bf6ee3700e34-serving-cert\") pod \"console-operator-7cd56b8dbc-qbz82\" (UID: \"ea9c808c-62b4-4dc7-9761-bf6ee3700e34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:06:43.629188 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:43.628966    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:06:43.629188 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.628987    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmhwk\" (UniqueName: \"kubernetes.io/projected/f14d2c99-6d70-44bf-9046-7cc1b84f1e7e-kube-api-access-tmhwk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-48hjx\" (UID: \"f14d2c99-6d70-44bf-9046-7cc1b84f1e7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-48hjx"
Oct 30 22:06:43.629188 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:43.629026    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea4b0705-2096-41da-b78b-b62590f236e8-networking-console-plugin-cert podName:ea4b0705-2096-41da-b78b-b62590f236e8 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:44.12900534 +0000 UTC m=+30.699098627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ea4b0705-2096-41da-b78b-b62590f236e8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-n42rm" (UID: "ea4b0705-2096-41da-b78b-b62590f236e8") : secret "networking-console-plugin-cert" not found
Oct 30 22:06:43.629188 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.629065    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b6651a05-7ba7-484f-8490-f22665bcb84f-image-registry-private-configuration\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:43.629188 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.629100    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ea4b0705-2096-41da-b78b-b62590f236e8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-n42rm\" (UID: \"ea4b0705-2096-41da-b78b-b62590f236e8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-n42rm"
Oct 30 22:06:43.629188 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.629124    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f14d2c99-6d70-44bf-9046-7cc1b84f1e7e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-48hjx\" (UID: \"f14d2c99-6d70-44bf-9046-7cc1b84f1e7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-48hjx"
Oct 30 22:06:43.629188 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.629166    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f14d2c99-6d70-44bf-9046-7cc1b84f1e7e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-48hjx\" (UID: \"f14d2c99-6d70-44bf-9046-7cc1b84f1e7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-48hjx"
Oct 30 22:06:43.629577 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.629197    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-bound-sa-token\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:43.629577 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.629265    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-certificates\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:43.629577 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.629285    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5b84639d-6b5d-4795-a356-8755ea234585-config\") pod \"service-ca-operator-544d749c8c-jd958\" (UID: \"5b84639d-6b5d-4795-a356-8755ea234585\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jd958"
Oct 30 22:06:43.629577 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.629297    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7gv7\" (UniqueName: \"kubernetes.io/projected/ea9c808c-62b4-4dc7-9761-bf6ee3700e34-kube-api-access-j7gv7\") pod \"console-operator-7cd56b8dbc-qbz82\" (UID: \"ea9c808c-62b4-4dc7-9761-bf6ee3700e34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:06:43.629577 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.629375    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vqtw4\" (UID: \"fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"
Oct 30 22:06:43.629577 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.629431    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b6651a05-7ba7-484f-8490-f22665bcb84f-installation-pull-secrets\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:43.629577 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.629537    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vqtw4\" (UID: \"fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"
Oct 30 22:06:43.629577 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.629564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ea9c808c-62b4-4dc7-9761-bf6ee3700e34-trusted-ca\") pod \"console-operator-7cd56b8dbc-qbz82\" (UID: \"ea9c808c-62b4-4dc7-9761-bf6ee3700e34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:06:43.629962 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.629631    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk8vc\" (UniqueName: \"kubernetes.io/projected/e0c1f2fe-ca1b-4556-8df1-134c67cee4f8-kube-api-access-lk8vc\") pod \"ingress-canary-p89nn\" (UID: \"e0c1f2fe-ca1b-4556-8df1-134c67cee4f8\") " pod="openshift-ingress-canary/ingress-canary-p89nn"
Oct 30 22:06:43.629962 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.629683    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b6651a05-7ba7-484f-8490-f22665bcb84f-ca-trust-extracted\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:43.629962 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.629709    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6651a05-7ba7-484f-8490-f22665bcb84f-trusted-ca\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:43.629962 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.629833    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ea9c808c-62b4-4dc7-9761-bf6ee3700e34-config\") pod \"console-operator-7cd56b8dbc-qbz82\" (UID: \"ea9c808c-62b4-4dc7-9761-bf6ee3700e34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:06:43.630155 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.630087    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f14d2c99-6d70-44bf-9046-7cc1b84f1e7e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-48hjx\" (UID: \"f14d2c99-6d70-44bf-9046-7cc1b84f1e7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-48hjx"
Oct 30 22:06:43.631187 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:43.630351    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:06:43.631187 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:43.630423    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls podName:fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a nodeName:}" failed. No retries permitted until 2026-10-30 22:06:44.130406982 +0000 UTC m=+30.700500258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vqtw4" (UID: "fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a") : secret "samples-operator-tls" not found
Oct 30 22:06:43.631187 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.630528    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ea4b0705-2096-41da-b78b-b62590f236e8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-n42rm\" (UID: \"ea4b0705-2096-41da-b78b-b62590f236e8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-n42rm"
Oct 30 22:06:43.631187 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.630613    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-certificates\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:43.631187 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.630745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5b84639d-6b5d-4795-a356-8755ea234585-serving-cert\") pod \"service-ca-operator-544d749c8c-jd958\" (UID: \"5b84639d-6b5d-4795-a356-8755ea234585\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jd958"
Oct 30 22:06:43.631187 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.630726    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6651a05-7ba7-484f-8490-f22665bcb84f-trusted-ca\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:43.631187 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.630819    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92nhs\" (UniqueName: \"kubernetes.io/projected/5b84639d-6b5d-4795-a356-8755ea234585-kube-api-access-92nhs\") pod \"service-ca-operator-544d749c8c-jd958\" (UID: \"5b84639d-6b5d-4795-a356-8755ea234585\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jd958"
Oct 30 22:06:43.631187 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.630871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bptmv\" (UniqueName: \"kubernetes.io/projected/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-kube-api-access-bptmv\") pod \"cluster-samples-operator-6ddb6c8b94-vqtw4\" (UID: \"fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"
Oct 30 22:06:43.631187 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.631189    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b6651a05-7ba7-484f-8490-f22665bcb84f-ca-trust-extracted\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:43.631840 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.631818    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ea9c808c-62b4-4dc7-9761-bf6ee3700e34-trusted-ca\") pod \"console-operator-7cd56b8dbc-qbz82\" (UID: \"ea9c808c-62b4-4dc7-9761-bf6ee3700e34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:06:43.632097 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.632078    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f14d2c99-6d70-44bf-9046-7cc1b84f1e7e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-48hjx\" (UID: \"f14d2c99-6d70-44bf-9046-7cc1b84f1e7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-48hjx"
Oct 30 22:06:43.632097 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.632090    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea9c808c-62b4-4dc7-9761-bf6ee3700e34-serving-cert\") pod \"console-operator-7cd56b8dbc-qbz82\" (UID: \"ea9c808c-62b4-4dc7-9761-bf6ee3700e34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:06:43.632499 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.632479    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0c1f2fe-ca1b-4556-8df1-134c67cee4f8-cert\") pod \"ingress-canary-p89nn\" (UID: \"e0c1f2fe-ca1b-4556-8df1-134c67cee4f8\") " pod="openshift-ingress-canary/ingress-canary-p89nn"
Oct 30 22:06:43.632564 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.632537    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b6651a05-7ba7-484f-8490-f22665bcb84f-image-registry-private-configuration\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:43.633444 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.633427    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5b84639d-6b5d-4795-a356-8755ea234585-serving-cert\") pod \"service-ca-operator-544d749c8c-jd958\" (UID: \"5b84639d-6b5d-4795-a356-8755ea234585\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jd958"
Oct 30 22:06:43.633755 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.633740    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b6651a05-7ba7-484f-8490-f22665bcb84f-installation-pull-secrets\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:43.643721 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.643696    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-dvcrc"
Oct 30 22:06:43.651569 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.651422    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-k8lw5"
Oct 30 22:06:43.651815 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.651791    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr8dq\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-kube-api-access-vr8dq\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:43.653079 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.653008    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h824x\" (UniqueName: \"kubernetes.io/projected/24c987a0-1cbf-4264-ba0e-586b738d08da-kube-api-access-h824x\") pod \"network-check-source-6f78d59f99-9xdrk\" (UID: \"24c987a0-1cbf-4264-ba0e-586b738d08da\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9xdrk"
Oct 30 22:06:43.653328 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.653305    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmhwk\" (UniqueName: \"kubernetes.io/projected/f14d2c99-6d70-44bf-9046-7cc1b84f1e7e-kube-api-access-tmhwk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-48hjx\" (UID: \"f14d2c99-6d70-44bf-9046-7cc1b84f1e7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-48hjx"
Oct 30 22:06:43.653413 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.653371    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-bound-sa-token\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:43.654758 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.654741    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bptmv\" (UniqueName: \"kubernetes.io/projected/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-kube-api-access-bptmv\") pod \"cluster-samples-operator-6ddb6c8b94-vqtw4\" (UID: \"fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"
Oct 30 22:06:43.654984 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.654963    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7gv7\" (UniqueName: \"kubernetes.io/projected/ea9c808c-62b4-4dc7-9761-bf6ee3700e34-kube-api-access-j7gv7\") pod \"console-operator-7cd56b8dbc-qbz82\" (UID: \"ea9c808c-62b4-4dc7-9761-bf6ee3700e34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:06:43.655028 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.655004    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92nhs\" (UniqueName: \"kubernetes.io/projected/5b84639d-6b5d-4795-a356-8755ea234585-kube-api-access-92nhs\") pod \"service-ca-operator-544d749c8c-jd958\" (UID: \"5b84639d-6b5d-4795-a356-8755ea234585\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jd958"
Oct 30 22:06:43.655113 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.655091    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk8vc\" (UniqueName: \"kubernetes.io/projected/e0c1f2fe-ca1b-4556-8df1-134c67cee4f8-kube-api-access-lk8vc\") pod \"ingress-canary-p89nn\" (UID: \"e0c1f2fe-ca1b-4556-8df1-134c67cee4f8\") " pod="openshift-ingress-canary/ingress-canary-p89nn"
Oct 30 22:06:43.661691 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.661669    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-74sl2"
Oct 30 22:06:43.682308 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.682276    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jh5sc"
Oct 30 22:06:43.718452 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.718413    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:43.731089 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.730263    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-48hjx"
Oct 30 22:06:43.739346 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.735690    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jd958"
Oct 30 22:06:43.742792 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.741412    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:06:43.753039 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.752452    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9xdrk"
Oct 30 22:06:43.753039 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.752872    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:43.768486 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.768321    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p89nn"
Oct 30 22:06:43.971508 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.971484    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dvcrc"]
Oct 30 22:06:43.974866 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:43.974838    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ee94be_1472_42e7_bf6e_d13e58671eeb.slice/crio-293e7aaa41bae7e7f8f02cc4f11203729a2c7a3b847ff7e0b29819b5671d3a83 WatchSource:0}: Error finding container 293e7aaa41bae7e7f8f02cc4f11203729a2c7a3b847ff7e0b29819b5671d3a83: Status 404 returned error can't find the container with id 293e7aaa41bae7e7f8f02cc4f11203729a2c7a3b847ff7e0b29819b5671d3a83
Oct 30 22:06:43.975452 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:43.975426    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f403b34_f155_49e6_b327_8ad89d00bbc3.slice/crio-784af23f4ecb79948e4642e53991b673c4b33f50f0bfb91aeb5ef7a88ca9ab7d WatchSource:0}: Error finding container 784af23f4ecb79948e4642e53991b673c4b33f50f0bfb91aeb5ef7a88ca9ab7d: Status 404 returned error can't find the container with id 784af23f4ecb79948e4642e53991b673c4b33f50f0bfb91aeb5ef7a88ca9ab7d
Oct 30 22:06:43.976404 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.976380    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-k8lw5"]
Oct 30 22:06:43.985112 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:43.985089    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-74sl2"]
Oct 30 22:06:43.988698 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:43.988673    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b74f51_6c71_45f2_97af_6aea20b9eeaf.slice/crio-3681ee3cc46929a2fb642aa02d8f7701443039483d0c39517a43ace21b066d3f WatchSource:0}: Error finding container 3681ee3cc46929a2fb642aa02d8f7701443039483d0c39517a43ace21b066d3f: Status 404 returned error can't find the container with id 3681ee3cc46929a2fb642aa02d8f7701443039483d0c39517a43ace21b066d3f
Oct 30 22:06:44.035675 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.035646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:44.035896 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.035719    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:44.035896 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.035764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qc4bw\" (UID: \"200052bd-2f0a-46d2-8b9b-6bd250b17b05\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"
Oct 30 22:06:44.035896 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.035819    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c31a736c-cd3e-42be-b783-fc25436b5302-metrics-tls\") pod \"dns-default-t2gq6\" (UID: \"c31a736c-cd3e-42be-b783-fc25436b5302\") " pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:44.035896 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.035845    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605c1b60-1fa5-4257-a588-4b2e0a78bcca-service-ca-bundle\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:44.036083 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:44.036045    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/605c1b60-1fa5-4257-a588-4b2e0a78bcca-service-ca-bundle podName:605c1b60-1fa5-4257-a588-4b2e0a78bcca nodeName:}" failed. No retries permitted until 2026-10-30 22:06:45.036026176 +0000 UTC m=+31.606119464 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/605c1b60-1fa5-4257-a588-4b2e0a78bcca-service-ca-bundle") pod "router-default-fb457cdb7-m99g8" (UID: "605c1b60-1fa5-4257-a588-4b2e0a78bcca") : configmap references non-existent config key: service-ca.crt
Oct 30 22:06:44.036389 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:44.036367    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:06:44.036389 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:44.036378    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:06:44.036538 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:44.036394    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:06:44.036538 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:44.036413    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54b978dcc8-sch2b: secret "image-registry-tls" not found
Oct 30 22:06:44.036538 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:44.036425    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls podName:200052bd-2f0a-46d2-8b9b-6bd250b17b05 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:45.03641082 +0000 UTC m=+31.606504110 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qc4bw" (UID: "200052bd-2f0a-46d2-8b9b-6bd250b17b05") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:06:44.036538 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:44.036445    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs podName:605c1b60-1fa5-4257-a588-4b2e0a78bcca nodeName:}" failed. No retries permitted until 2026-10-30 22:06:45.03643599 +0000 UTC m=+31.606529264 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs") pod "router-default-fb457cdb7-m99g8" (UID: "605c1b60-1fa5-4257-a588-4b2e0a78bcca") : secret "router-metrics-certs-default" not found
Oct 30 22:06:44.036538 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:44.036469    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls podName:e297483a-0b86-4481-9597-0c89e9a6a7ac nodeName:}" failed. No retries permitted until 2026-10-30 22:06:45.036450526 +0000 UTC m=+31.606543822 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls") pod "image-registry-54b978dcc8-sch2b" (UID: "e297483a-0b86-4481-9597-0c89e9a6a7ac") : secret "image-registry-tls" not found
Oct 30 22:06:44.036538 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:44.036495    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:06:44.036538 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:44.036541    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c31a736c-cd3e-42be-b783-fc25436b5302-metrics-tls podName:c31a736c-cd3e-42be-b783-fc25436b5302 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:45.036527058 +0000 UTC m=+31.606620346 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c31a736c-cd3e-42be-b783-fc25436b5302-metrics-tls") pod "dns-default-t2gq6" (UID: "c31a736c-cd3e-42be-b783-fc25436b5302") : secret "dns-default-metrics-tls" not found
Oct 30 22:06:44.041107 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.041079    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jh5sc"]
Oct 30 22:06:44.042550 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:44.042523    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88fe4dfd_dcda_444d_871b_24ca3fb10d77.slice/crio-4f2b127804d27f49d0c9bed28f086e3bb999bbd9fa415e695653470e9034f375 WatchSource:0}: Error finding container 4f2b127804d27f49d0c9bed28f086e3bb999bbd9fa415e695653470e9034f375: Status 404 returned error can't find the container with id 4f2b127804d27f49d0c9bed28f086e3bb999bbd9fa415e695653470e9034f375
Oct 30 22:06:44.109027 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.109005    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-jd958"]
Oct 30 22:06:44.110508 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.110489    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-qbz82"]
Oct 30 22:06:44.112041 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:44.112018    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea9c808c_62b4_4dc7_9761_bf6ee3700e34.slice/crio-da6d5bc0a53d2d71c94f4708be735cb287c3b7824e15690d1f3472fb87cb3453 WatchSource:0}: Error finding container da6d5bc0a53d2d71c94f4708be735cb287c3b7824e15690d1f3472fb87cb3453: Status 404 returned error can't find the container with id da6d5bc0a53d2d71c94f4708be735cb287c3b7824e15690d1f3472fb87cb3453
Oct 30 22:06:44.113026 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.113004    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-48hjx"]
Oct 30 22:06:44.116860 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:44.116736    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14d2c99_6d70_44bf_9046_7cc1b84f1e7e.slice/crio-4faf0685059f939c00321305f975604de2c8a02d6a234aa7fb7bd4792e0661d2 WatchSource:0}: Error finding container 4faf0685059f939c00321305f975604de2c8a02d6a234aa7fb7bd4792e0661d2: Status 404 returned error can't find the container with id 4faf0685059f939c00321305f975604de2c8a02d6a234aa7fb7bd4792e0661d2
Oct 30 22:06:44.126745 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.126613    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-vwshh"]
Oct 30 22:06:44.137059 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.136538    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:44.137059 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.136607    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea4b0705-2096-41da-b78b-b62590f236e8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-n42rm\" (UID: \"ea4b0705-2096-41da-b78b-b62590f236e8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-n42rm"
Oct 30 22:06:44.137059 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.136692    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vqtw4\" (UID: \"fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"
Oct 30 22:06:44.137059 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:44.136843    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:06:44.137059 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:44.136855    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:06:44.137059 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:44.136884    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b9d7c7dff-82x2r: secret "image-registry-tls" not found
Oct 30 22:06:44.137059 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:44.136902    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls podName:fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a nodeName:}" failed. No retries permitted until 2026-10-30 22:06:45.136882247 +0000 UTC m=+31.706975541 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vqtw4" (UID: "fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a") : secret "samples-operator-tls" not found
Oct 30 22:06:44.137059 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:44.136945    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls podName:b6651a05-7ba7-484f-8490-f22665bcb84f nodeName:}" failed. No retries permitted until 2026-10-30 22:06:45.136927946 +0000 UTC m=+31.707021232 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls") pod "image-registry-6b9d7c7dff-82x2r" (UID: "b6651a05-7ba7-484f-8490-f22665bcb84f") : secret "image-registry-tls" not found
Oct 30 22:06:44.137059 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:44.136999    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:06:44.137059 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:44.137038    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea4b0705-2096-41da-b78b-b62590f236e8-networking-console-plugin-cert podName:ea4b0705-2096-41da-b78b-b62590f236e8 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:45.137025974 +0000 UTC m=+31.707119261 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ea4b0705-2096-41da-b78b-b62590f236e8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-n42rm" (UID: "ea4b0705-2096-41da-b78b-b62590f236e8") : secret "networking-console-plugin-cert" not found
Oct 30 22:06:44.137602 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.137468    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-9xdrk"]
Oct 30 22:06:44.149407 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.149382    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jd958" event={"ID":"5b84639d-6b5d-4795-a356-8755ea234585","Type":"ContainerStarted","Data":"c543d14496ba1d58230b67dac2cad4b80d74f21e802d0bc59043dfd072cbcee4"}
Oct 30 22:06:44.151375 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.151335    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-74sl2" event={"ID":"00b74f51-6c71-45f2-97af-6aea20b9eeaf","Type":"ContainerStarted","Data":"6d573af53ca384993765b9f7cef581e859d0d7ff9e6a34e6c699592a4b5bf387"}
Oct 30 22:06:44.151468 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.151380    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-74sl2" event={"ID":"00b74f51-6c71-45f2-97af-6aea20b9eeaf","Type":"ContainerStarted","Data":"3681ee3cc46929a2fb642aa02d8f7701443039483d0c39517a43ace21b066d3f"}
Oct 30 22:06:44.152489 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.152466    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9xdrk" event={"ID":"24c987a0-1cbf-4264-ba0e-586b738d08da","Type":"ContainerStarted","Data":"aa6e8f0edb6759c894052cf03580233f00ba2d529c93f92e36a4b1b9efe9f64b"}
Oct 30 22:06:44.153559 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.153535    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82" event={"ID":"ea9c808c-62b4-4dc7-9761-bf6ee3700e34","Type":"ContainerStarted","Data":"da6d5bc0a53d2d71c94f4708be735cb287c3b7824e15690d1f3472fb87cb3453"}
Oct 30 22:06:44.154671 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.154652    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jh5sc" event={"ID":"88fe4dfd-dcda-444d-871b-24ca3fb10d77","Type":"ContainerStarted","Data":"4f2b127804d27f49d0c9bed28f086e3bb999bbd9fa415e695653470e9034f375"}
Oct 30 22:06:44.156105 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.156082    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-dvcrc" event={"ID":"b8ee94be-1472-42e7-bf6e-d13e58671eeb","Type":"ContainerStarted","Data":"5cb80d5ac1de17a135d786d316be365669b3eeb05d262a84e6a8aad517dacc67"}
Oct 30 22:06:44.156186 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.156123    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-dvcrc" event={"ID":"b8ee94be-1472-42e7-bf6e-d13e58671eeb","Type":"ContainerStarted","Data":"293e7aaa41bae7e7f8f02cc4f11203729a2c7a3b847ff7e0b29819b5671d3a83"}
Oct 30 22:06:44.157265 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.157244    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-48hjx" event={"ID":"f14d2c99-6d70-44bf-9046-7cc1b84f1e7e","Type":"ContainerStarted","Data":"4faf0685059f939c00321305f975604de2c8a02d6a234aa7fb7bd4792e0661d2"}
Oct 30 22:06:44.158372 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.158351    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-vwshh" event={"ID":"965e7b7f-9044-4c69-be4d-06ef9a1044e2","Type":"ContainerStarted","Data":"471d2921a86a8eaa2f2bb973796671cf840e2862e459183ac3d4afe68995188b"}
Oct 30 22:06:44.159688 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.159665    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-k8lw5" event={"ID":"3f403b34-f155-49e6-b327-8ad89d00bbc3","Type":"ContainerStarted","Data":"ca2ef5dd42874b4fd178ff51ac0c32adffcf1832bd0ffdbbc61c5e43ba122867"}
Oct 30 22:06:44.159785 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.159706    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-k8lw5" event={"ID":"3f403b34-f155-49e6-b327-8ad89d00bbc3","Type":"ContainerStarted","Data":"784af23f4ecb79948e4642e53991b673c4b33f50f0bfb91aeb5ef7a88ca9ab7d"}
Oct 30 22:06:44.179907 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.179889    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p89nn"]
Oct 30 22:06:44.181460 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:44.181431    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c1f2fe_ca1b_4556_8df1_134c67cee4f8.slice/crio-7d1343f6fe0cbba61605e7232037aa2743561125a07d7556dbdfd92daed9210c WatchSource:0}: Error finding container 7d1343f6fe0cbba61605e7232037aa2743561125a07d7556dbdfd92daed9210c: Status 404 returned error can't find the container with id 7d1343f6fe0cbba61605e7232037aa2743561125a07d7556dbdfd92daed9210c
Oct 30 22:06:44.192354 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.192316    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-74sl2" podStartSLOduration=352.192304565 podStartE2EDuration="5m52.192304565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:44.191682433 +0000 UTC m=+30.761775728" watchObservedRunningTime="2026-10-30 22:06:44.192304565 +0000 UTC m=+30.762397853"
Oct 30 22:06:44.233654 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.233566    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-dvcrc" podStartSLOduration=352.233554317 podStartE2EDuration="5m52.233554317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:44.233114556 +0000 UTC m=+30.803207850" watchObservedRunningTime="2026-10-30 22:06:44.233554317 +0000 UTC m=+30.803647609"
Oct 30 22:06:44.274640 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.274584    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-k8lw5" podStartSLOduration=352.274571184 podStartE2EDuration="5m52.274571184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:44.27432697 +0000 UTC m=+30.844420265" watchObservedRunningTime="2026-10-30 22:06:44.274571184 +0000 UTC m=+30.844664469"
Oct 30 22:06:44.772089 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.772053    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:44.874485 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.874456    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:44.875013 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.874986    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:44.875505 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.875483    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:44.892890 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.892854    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ggpvh\""
Oct 30 22:06:44.893036 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.892930    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 22:06:44.893036 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.893033    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 22:06:44.893175 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.893160    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-bf7kf\""
Oct 30 22:06:44.893334 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:44.893317    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c5qz2\""
Oct 30 22:06:45.045917 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:45.045834    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:45.046396 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:45.045918    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:45.046396 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:45.045986    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qc4bw\" (UID: \"200052bd-2f0a-46d2-8b9b-6bd250b17b05\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"
Oct 30 22:06:45.046396 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:45.046049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c31a736c-cd3e-42be-b783-fc25436b5302-metrics-tls\") pod \"dns-default-t2gq6\" (UID: \"c31a736c-cd3e-42be-b783-fc25436b5302\") " pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:45.046396 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:45.046090    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605c1b60-1fa5-4257-a588-4b2e0a78bcca-service-ca-bundle\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:45.046396 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:45.046341    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/605c1b60-1fa5-4257-a588-4b2e0a78bcca-service-ca-bundle podName:605c1b60-1fa5-4257-a588-4b2e0a78bcca nodeName:}" failed. No retries permitted until 2026-10-30 22:06:47.046322022 +0000 UTC m=+33.616415309 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/605c1b60-1fa5-4257-a588-4b2e0a78bcca-service-ca-bundle") pod "router-default-fb457cdb7-m99g8" (UID: "605c1b60-1fa5-4257-a588-4b2e0a78bcca") : configmap references non-existent config key: service-ca.crt
Oct 30 22:06:45.046784 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:45.046762    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:06:45.046883 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:45.046820    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs podName:605c1b60-1fa5-4257-a588-4b2e0a78bcca nodeName:}" failed. No retries permitted until 2026-10-30 22:06:47.04680552 +0000 UTC m=+33.616898806 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs") pod "router-default-fb457cdb7-m99g8" (UID: "605c1b60-1fa5-4257-a588-4b2e0a78bcca") : secret "router-metrics-certs-default" not found
Oct 30 22:06:45.046950 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:45.046890    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:06:45.046950 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:45.046901    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54b978dcc8-sch2b: secret "image-registry-tls" not found
Oct 30 22:06:45.046950 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:45.046937    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls podName:e297483a-0b86-4481-9597-0c89e9a6a7ac nodeName:}" failed. No retries permitted until 2026-10-30 22:06:47.046925449 +0000 UTC m=+33.617018736 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls") pod "image-registry-54b978dcc8-sch2b" (UID: "e297483a-0b86-4481-9597-0c89e9a6a7ac") : secret "image-registry-tls" not found
Oct 30 22:06:45.047430 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:45.047410    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:06:45.047512 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:45.047464    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls podName:200052bd-2f0a-46d2-8b9b-6bd250b17b05 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:47.047449689 +0000 UTC m=+33.617542968 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qc4bw" (UID: "200052bd-2f0a-46d2-8b9b-6bd250b17b05") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:06:45.047574 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:45.047528    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:06:45.047574 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:45.047563    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c31a736c-cd3e-42be-b783-fc25436b5302-metrics-tls podName:c31a736c-cd3e-42be-b783-fc25436b5302 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:47.047551772 +0000 UTC m=+33.617645065 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c31a736c-cd3e-42be-b783-fc25436b5302-metrics-tls") pod "dns-default-t2gq6" (UID: "c31a736c-cd3e-42be-b783-fc25436b5302") : secret "dns-default-metrics-tls" not found
Oct 30 22:06:45.150774 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:45.150742    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea4b0705-2096-41da-b78b-b62590f236e8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-n42rm\" (UID: \"ea4b0705-2096-41da-b78b-b62590f236e8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-n42rm"
Oct 30 22:06:45.150955 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:45.150836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vqtw4\" (UID: \"fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"
Oct 30 22:06:45.151021 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:45.150963    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:45.151324 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:45.151116    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:06:45.151324 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:45.151134    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b9d7c7dff-82x2r: secret "image-registry-tls" not found
Oct 30 22:06:45.151324 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:45.151195    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:06:45.151324 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:45.151200    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls podName:b6651a05-7ba7-484f-8490-f22665bcb84f nodeName:}" failed. No retries permitted until 2026-10-30 22:06:47.151181478 +0000 UTC m=+33.721274774 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls") pod "image-registry-6b9d7c7dff-82x2r" (UID: "b6651a05-7ba7-484f-8490-f22665bcb84f") : secret "image-registry-tls" not found
Oct 30 22:06:45.151324 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:45.151195    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:06:45.151324 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:45.151280    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea4b0705-2096-41da-b78b-b62590f236e8-networking-console-plugin-cert podName:ea4b0705-2096-41da-b78b-b62590f236e8 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:47.151262766 +0000 UTC m=+33.721356054 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ea4b0705-2096-41da-b78b-b62590f236e8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-n42rm" (UID: "ea4b0705-2096-41da-b78b-b62590f236e8") : secret "networking-console-plugin-cert" not found
Oct 30 22:06:45.151324 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:45.151300    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls podName:fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a nodeName:}" failed. No retries permitted until 2026-10-30 22:06:47.151290382 +0000 UTC m=+33.721383668 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vqtw4" (UID: "fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a") : secret "samples-operator-tls" not found
Oct 30 22:06:45.166131 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:45.166094    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p89nn" event={"ID":"e0c1f2fe-ca1b-4556-8df1-134c67cee4f8","Type":"ContainerStarted","Data":"7d1343f6fe0cbba61605e7232037aa2743561125a07d7556dbdfd92daed9210c"}
Oct 30 22:06:45.994958 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:45.994931    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:46.173036 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:46.172988    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:46.173892 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:46.173863    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jh5sc" event={"ID":"88fe4dfd-dcda-444d-871b-24ca3fb10d77","Type":"ContainerStarted","Data":"7c0623d54b1b4a7bed5e28b2bca191587d8cafa7eda77ff848a5a75ff7eabc8b"}
Oct 30 22:06:46.220006 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:46.219899    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jh5sc" podStartSLOduration=346.322721597 podStartE2EDuration="5m48.219879111s" totalImagesPullingTime="1.897157514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:46.219185641 +0000 UTC m=+32.789278938" watchObservedRunningTime="2026-10-30 22:06:46.219879111 +0000 UTC m=+32.789972407"
Oct 30 22:06:46.394243 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:46.394185    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:46.563487 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:46.563449    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:47.079751 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:47.077673    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qc4bw\" (UID: \"200052bd-2f0a-46d2-8b9b-6bd250b17b05\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"
Oct 30 22:06:47.079751 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:47.077754    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c31a736c-cd3e-42be-b783-fc25436b5302-metrics-tls\") pod \"dns-default-t2gq6\" (UID: \"c31a736c-cd3e-42be-b783-fc25436b5302\") " pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:47.079751 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:47.077787    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605c1b60-1fa5-4257-a588-4b2e0a78bcca-service-ca-bundle\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:47.079751 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:47.077880    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:47.079751 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:47.077937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:47.079751 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:47.078070    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:06:47.079751 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:47.078084    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54b978dcc8-sch2b: secret "image-registry-tls" not found
Oct 30 22:06:47.079751 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:47.078140    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls podName:e297483a-0b86-4481-9597-0c89e9a6a7ac nodeName:}" failed. No retries permitted until 2026-10-30 22:06:51.07812401 +0000 UTC m=+37.648217300 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls") pod "image-registry-54b978dcc8-sch2b" (UID: "e297483a-0b86-4481-9597-0c89e9a6a7ac") : secret "image-registry-tls" not found
Oct 30 22:06:47.079751 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:47.078583    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:06:47.079751 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:47.078634    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls podName:200052bd-2f0a-46d2-8b9b-6bd250b17b05 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:51.078618129 +0000 UTC m=+37.648711410 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qc4bw" (UID: "200052bd-2f0a-46d2-8b9b-6bd250b17b05") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:06:47.079751 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:47.078686    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:06:47.079751 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:47.078715    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c31a736c-cd3e-42be-b783-fc25436b5302-metrics-tls podName:c31a736c-cd3e-42be-b783-fc25436b5302 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:51.078705736 +0000 UTC m=+37.648799014 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c31a736c-cd3e-42be-b783-fc25436b5302-metrics-tls") pod "dns-default-t2gq6" (UID: "c31a736c-cd3e-42be-b783-fc25436b5302") : secret "dns-default-metrics-tls" not found
Oct 30 22:06:47.079751 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:47.078774    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/605c1b60-1fa5-4257-a588-4b2e0a78bcca-service-ca-bundle podName:605c1b60-1fa5-4257-a588-4b2e0a78bcca nodeName:}" failed. No retries permitted until 2026-10-30 22:06:51.078764669 +0000 UTC m=+37.648857957 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/605c1b60-1fa5-4257-a588-4b2e0a78bcca-service-ca-bundle") pod "router-default-fb457cdb7-m99g8" (UID: "605c1b60-1fa5-4257-a588-4b2e0a78bcca") : configmap references non-existent config key: service-ca.crt
Oct 30 22:06:47.079751 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:47.078825    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:06:47.079751 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:47.078852    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs podName:605c1b60-1fa5-4257-a588-4b2e0a78bcca nodeName:}" failed. No retries permitted until 2026-10-30 22:06:51.078841706 +0000 UTC m=+37.648934994 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs") pod "router-default-fb457cdb7-m99g8" (UID: "605c1b60-1fa5-4257-a588-4b2e0a78bcca") : secret "router-metrics-certs-default" not found
Oct 30 22:06:47.178969 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:47.178513    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vqtw4\" (UID: \"fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"
Oct 30 22:06:47.178969 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:47.178643    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:47.178969 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:47.178697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea4b0705-2096-41da-b78b-b62590f236e8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-n42rm\" (UID: \"ea4b0705-2096-41da-b78b-b62590f236e8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-n42rm"
Oct 30 22:06:47.178969 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:47.178718    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:06:47.178969 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:47.178793    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls podName:fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a nodeName:}" failed. No retries permitted until 2026-10-30 22:06:51.178772724 +0000 UTC m=+37.748866012 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vqtw4" (UID: "fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a") : secret "samples-operator-tls" not found
Oct 30 22:06:47.178969 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:47.178820    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:06:47.178969 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:47.178831    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:06:47.178969 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:47.178839    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b9d7c7dff-82x2r: secret "image-registry-tls" not found
Oct 30 22:06:47.178969 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:47.178883    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea4b0705-2096-41da-b78b-b62590f236e8-networking-console-plugin-cert podName:ea4b0705-2096-41da-b78b-b62590f236e8 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:51.178869481 +0000 UTC m=+37.748962776 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ea4b0705-2096-41da-b78b-b62590f236e8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-n42rm" (UID: "ea4b0705-2096-41da-b78b-b62590f236e8") : secret "networking-console-plugin-cert" not found
Oct 30 22:06:47.178969 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:47.178900    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls podName:b6651a05-7ba7-484f-8490-f22665bcb84f nodeName:}" failed. No retries permitted until 2026-10-30 22:06:51.178891349 +0000 UTC m=+37.748984623 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls") pod "image-registry-6b9d7c7dff-82x2r" (UID: "b6651a05-7ba7-484f-8490-f22665bcb84f") : secret "image-registry-tls" not found
Oct 30 22:06:47.684990 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:47.684047    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e444015-5346-4106-93aa-ef43eb9d0be6-original-pull-secret\") pod \"global-pull-secret-syncer-ljnhp\" (UID: \"1e444015-5346-4106-93aa-ef43eb9d0be6\") " pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:47.684990 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:47.684126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs\") pod \"network-metrics-daemon-fvsmv\" (UID: \"4049e53b-bffe-40fb-beef-22acc171315a\") " pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:47.684990 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:47.684308    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:06:47.684990 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:47.684373    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs podName:4049e53b-bffe-40fb-beef-22acc171315a nodeName:}" failed. No retries permitted until 2026-10-30 22:07:19.684353915 +0000 UTC m=+66.254447217 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs") pod "network-metrics-daemon-fvsmv" (UID: "4049e53b-bffe-40fb-beef-22acc171315a") : secret "metrics-daemon-secret" not found
Oct 30 22:06:47.690958 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:47.690929    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e444015-5346-4106-93aa-ef43eb9d0be6-original-pull-secret\") pod \"global-pull-secret-syncer-ljnhp\" (UID: \"1e444015-5346-4106-93aa-ef43eb9d0be6\") " pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:47.785517 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:47.785479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wthz4\" (UniqueName: \"kubernetes.io/projected/d303600a-2390-4ba9-8ea8-a58bd505a109-kube-api-access-wthz4\") pod \"network-check-target-zcl2w\" (UID: \"d303600a-2390-4ba9-8ea8-a58bd505a109\") " pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:47.788932 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:47.788903    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wthz4\" (UniqueName: \"kubernetes.io/projected/d303600a-2390-4ba9-8ea8-a58bd505a109-kube-api-access-wthz4\") pod \"network-check-target-zcl2w\" (UID: \"d303600a-2390-4ba9-8ea8-a58bd505a109\") " pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:47.919388 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:47.919355    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:47.937906 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:47.937836    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:48.453367 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:48.453336    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ljnhp"]
Oct 30 22:06:48.463648 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:48.463617    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zcl2w"]
Oct 30 22:06:48.576125 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:48.576084    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e444015_5346_4106_93aa_ef43eb9d0be6.slice/crio-eac8bfd37e67bf7de6667108361618f1e829410e6a381e7b137b3781e55f3ce4 WatchSource:0}: Error finding container eac8bfd37e67bf7de6667108361618f1e829410e6a381e7b137b3781e55f3ce4: Status 404 returned error can't find the container with id eac8bfd37e67bf7de6667108361618f1e829410e6a381e7b137b3781e55f3ce4
Oct 30 22:06:48.577060 ip-10-0-142-185 kubenswrapper[2580]: W1030 22:06:48.577041    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd303600a_2390_4ba9_8ea8_a58bd505a109.slice/crio-473049b96ecb4925bb44684d88d7ed4a537e37099bdd407ebe98583432f67887 WatchSource:0}: Error finding container 473049b96ecb4925bb44684d88d7ed4a537e37099bdd407ebe98583432f67887: Status 404 returned error can't find the container with id 473049b96ecb4925bb44684d88d7ed4a537e37099bdd407ebe98583432f67887
Oct 30 22:06:49.189371 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:49.188321    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/0.log"
Oct 30 22:06:49.189371 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:49.188361    2580 generic.go:356] "Generic (PLEG): container finished" podID="ea9c808c-62b4-4dc7-9761-bf6ee3700e34" containerID="52cf9adc07a711d05aeb39d35a56ea1885f30e3391aca7022ae6205d95024486" exitCode=255
Oct 30 22:06:49.189371 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:49.188456    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82" event={"ID":"ea9c808c-62b4-4dc7-9761-bf6ee3700e34","Type":"ContainerDied","Data":"52cf9adc07a711d05aeb39d35a56ea1885f30e3391aca7022ae6205d95024486"}
Oct 30 22:06:49.189371 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:49.188800    2580 scope.go:122] "RemoveContainer" containerID="52cf9adc07a711d05aeb39d35a56ea1885f30e3391aca7022ae6205d95024486"
Oct 30 22:06:49.191955 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:49.191564    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-48hjx" event={"ID":"f14d2c99-6d70-44bf-9046-7cc1b84f1e7e","Type":"ContainerStarted","Data":"2d7e8d26fe3fd8f53c8f4bd67e9072719d373db10007cf1a6d23b1a0dedd9ac7"}
Oct 30 22:06:49.193238 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:49.193185    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-vwshh" event={"ID":"965e7b7f-9044-4c69-be4d-06ef9a1044e2","Type":"ContainerStarted","Data":"afd146d373d52e0f98ce65ee914e9fbfa8ccd00e7769f91d073a424aafe5ef2f"}
Oct 30 22:06:49.194875 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:49.194849    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ljnhp" event={"ID":"1e444015-5346-4106-93aa-ef43eb9d0be6","Type":"ContainerStarted","Data":"eac8bfd37e67bf7de6667108361618f1e829410e6a381e7b137b3781e55f3ce4"}
Oct 30 22:06:49.196372 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:49.196346    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zcl2w" event={"ID":"d303600a-2390-4ba9-8ea8-a58bd505a109","Type":"ContainerStarted","Data":"473049b96ecb4925bb44684d88d7ed4a537e37099bdd407ebe98583432f67887"}
Oct 30 22:06:49.198964 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:49.198199    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jd958" event={"ID":"5b84639d-6b5d-4795-a356-8755ea234585","Type":"ContainerStarted","Data":"04afb2fe118c5022d23bd3a0737d555ce31d496ab0eabfe2c30ecf521f94ca48"}
Oct 30 22:06:49.282056 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:49.282003    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jd958" podStartSLOduration=322.580222128 podStartE2EDuration="5m27.281988469s" totalImagesPullingTime="4.701766341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:49.281382022 +0000 UTC m=+35.851475317" watchObservedRunningTime="2026-10-30 22:06:49.281988469 +0000 UTC m=+35.852081763"
Oct 30 22:06:49.329347 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:49.326446    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-48hjx" podStartSLOduration=322.643582037 podStartE2EDuration="5m27.326427591s" totalImagesPullingTime="4.682845554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:49.3258779 +0000 UTC m=+35.895971221" watchObservedRunningTime="2026-10-30 22:06:49.326427591 +0000 UTC m=+35.896520887"
Oct 30 22:06:49.381013 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:49.380606    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-vwshh" podStartSLOduration=322.697637981 podStartE2EDuration="5m27.380581938s" totalImagesPullingTime="4.682943957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:49.371600348 +0000 UTC m=+35.941693644" watchObservedRunningTime="2026-10-30 22:06:49.380581938 +0000 UTC m=+35.950675235"
Oct 30 22:06:50.084914 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.084875    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mwjsl"]
Oct 30 22:06:50.100363 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.100326    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:50.109026 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.108946    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mwjsl"]
Oct 30 22:06:50.123499 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.123451    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 22:06:50.123907 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.123869    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-26rrv\""
Oct 30 22:06:50.160185 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.160138    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 22:06:50.208283 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.208181    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p89nn" event={"ID":"e0c1f2fe-ca1b-4556-8df1-134c67cee4f8","Type":"ContainerStarted","Data":"f2cbc0bd33186883dc812be38de5d430dd4c3777c02fb32fdc316c8a85e63e29"}
Oct 30 22:06:50.211684 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.211653    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/052e2399-f78d-4eb2-9da7-e2b098f86da7-crio-socket\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:50.212135 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.211810    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/052e2399-f78d-4eb2-9da7-e2b098f86da7-data-volume\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:50.212135 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.211848    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5cps\" (UniqueName: \"kubernetes.io/projected/052e2399-f78d-4eb2-9da7-e2b098f86da7-kube-api-access-q5cps\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:50.212135 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.211923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:50.212135 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.211956    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/052e2399-f78d-4eb2-9da7-e2b098f86da7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:50.212850 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.212828    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/1.log"
Oct 30 22:06:50.213534 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.213514    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/0.log"
Oct 30 22:06:50.213646 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.213556    2580 generic.go:356] "Generic (PLEG): container finished" podID="ea9c808c-62b4-4dc7-9761-bf6ee3700e34" containerID="d60d0383fb24f739b8698d09dfc099d55168ef8507ac722ceb2547384d90c615" exitCode=255
Oct 30 22:06:50.213851 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.213821    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82" event={"ID":"ea9c808c-62b4-4dc7-9761-bf6ee3700e34","Type":"ContainerDied","Data":"d60d0383fb24f739b8698d09dfc099d55168ef8507ac722ceb2547384d90c615"}
Oct 30 22:06:50.213924 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.213888    2580 scope.go:122] "RemoveContainer" containerID="52cf9adc07a711d05aeb39d35a56ea1885f30e3391aca7022ae6205d95024486"
Oct 30 22:06:50.214314 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.214298    2580 scope.go:122] "RemoveContainer" containerID="d60d0383fb24f739b8698d09dfc099d55168ef8507ac722ceb2547384d90c615"
Oct 30 22:06:50.214497 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:50.214473    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qbz82_openshift-console-operator(ea9c808c-62b4-4dc7-9761-bf6ee3700e34)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82" podUID="ea9c808c-62b4-4dc7-9761-bf6ee3700e34"
Oct 30 22:06:50.219192 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.219167    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:50.252238 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.252184    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-p89nn" podStartSLOduration=2.142747611 podStartE2EDuration="7.252169232s" totalImagesPullingTime="5.109421621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:50.251489359 +0000 UTC m=+36.821582653" watchObservedRunningTime="2026-10-30 22:06:50.252169232 +0000 UTC m=+36.822262526"
Oct 30 22:06:50.313547 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.312884    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:50.313547 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.312932    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/052e2399-f78d-4eb2-9da7-e2b098f86da7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:50.313547 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.313161    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/052e2399-f78d-4eb2-9da7-e2b098f86da7-crio-socket\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:50.313547 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.313357    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/052e2399-f78d-4eb2-9da7-e2b098f86da7-data-volume\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:50.313547 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.313400    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5cps\" (UniqueName: \"kubernetes.io/projected/052e2399-f78d-4eb2-9da7-e2b098f86da7-kube-api-access-q5cps\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:50.313952 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:50.313869    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:50.313952 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:50.313928    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls podName:052e2399-f78d-4eb2-9da7-e2b098f86da7 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:50.813908734 +0000 UTC m=+37.384002010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mwjsl" (UID: "052e2399-f78d-4eb2-9da7-e2b098f86da7") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:50.314064 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:50.313965    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 22:06:50.314064 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:50.313986    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/052e2399-f78d-4eb2-9da7-e2b098f86da7-kube-rbac-proxy-cm podName:052e2399-f78d-4eb2-9da7-e2b098f86da7 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:50.813980277 +0000 UTC m=+37.384073551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/052e2399-f78d-4eb2-9da7-e2b098f86da7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mwjsl" (UID: "052e2399-f78d-4eb2-9da7-e2b098f86da7") : configmap "kube-rbac-proxy" not found
Oct 30 22:06:50.315379 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.315348    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/052e2399-f78d-4eb2-9da7-e2b098f86da7-crio-socket\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:50.315559 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.315539    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/052e2399-f78d-4eb2-9da7-e2b098f86da7-data-volume\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:50.341896 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.341827    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5cps\" (UniqueName: \"kubernetes.io/projected/052e2399-f78d-4eb2-9da7-e2b098f86da7-kube-api-access-q5cps\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:50.402491 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.402453    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:50.574740 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.574659    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:50.746045 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.746009    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:50.818251 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.817558    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:50.818251 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.817612    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/052e2399-f78d-4eb2-9da7-e2b098f86da7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:50.818251 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:50.817742    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:50.818251 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:50.817816    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls podName:052e2399-f78d-4eb2-9da7-e2b098f86da7 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:51.817796015 +0000 UTC m=+38.387889295 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mwjsl" (UID: "052e2399-f78d-4eb2-9da7-e2b098f86da7") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:50.818251 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:50.817848    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 22:06:50.818251 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:50.817894    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/052e2399-f78d-4eb2-9da7-e2b098f86da7-kube-rbac-proxy-cm podName:052e2399-f78d-4eb2-9da7-e2b098f86da7 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:51.817883492 +0000 UTC m=+38.387976774 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/052e2399-f78d-4eb2-9da7-e2b098f86da7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mwjsl" (UID: "052e2399-f78d-4eb2-9da7-e2b098f86da7") : configmap "kube-rbac-proxy" not found
Oct 30 22:06:50.968517 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:50.968466    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:51.121020 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:51.120939    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c31a736c-cd3e-42be-b783-fc25436b5302-metrics-tls\") pod \"dns-default-t2gq6\" (UID: \"c31a736c-cd3e-42be-b783-fc25436b5302\") " pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:51.121020 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:51.120982    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605c1b60-1fa5-4257-a588-4b2e0a78bcca-service-ca-bundle\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:51.124480 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:51.121080    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:51.124480 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:51.121178    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:51.124480 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:51.121253    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qc4bw\" (UID: \"200052bd-2f0a-46d2-8b9b-6bd250b17b05\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"
Oct 30 22:06:51.124480 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:51.121392    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:06:51.124480 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:51.121513    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls podName:200052bd-2f0a-46d2-8b9b-6bd250b17b05 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:59.121492953 +0000 UTC m=+45.691586241 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qc4bw" (UID: "200052bd-2f0a-46d2-8b9b-6bd250b17b05") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:06:51.124480 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:51.121917    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:06:51.124480 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:51.121968    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c31a736c-cd3e-42be-b783-fc25436b5302-metrics-tls podName:c31a736c-cd3e-42be-b783-fc25436b5302 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:59.121952528 +0000 UTC m=+45.692045815 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c31a736c-cd3e-42be-b783-fc25436b5302-metrics-tls") pod "dns-default-t2gq6" (UID: "c31a736c-cd3e-42be-b783-fc25436b5302") : secret "dns-default-metrics-tls" not found
Oct 30 22:06:51.124480 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:51.122035    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/605c1b60-1fa5-4257-a588-4b2e0a78bcca-service-ca-bundle podName:605c1b60-1fa5-4257-a588-4b2e0a78bcca nodeName:}" failed. No retries permitted until 2026-10-30 22:06:59.122023803 +0000 UTC m=+45.692117078 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/605c1b60-1fa5-4257-a588-4b2e0a78bcca-service-ca-bundle") pod "router-default-fb457cdb7-m99g8" (UID: "605c1b60-1fa5-4257-a588-4b2e0a78bcca") : configmap references non-existent config key: service-ca.crt
Oct 30 22:06:51.124480 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:51.122087    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:06:51.124480 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:51.122117    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs podName:605c1b60-1fa5-4257-a588-4b2e0a78bcca nodeName:}" failed. No retries permitted until 2026-10-30 22:06:59.122107071 +0000 UTC m=+45.692200358 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs") pod "router-default-fb457cdb7-m99g8" (UID: "605c1b60-1fa5-4257-a588-4b2e0a78bcca") : secret "router-metrics-certs-default" not found
Oct 30 22:06:51.124480 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:51.122171    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:06:51.124480 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:51.122181    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54b978dcc8-sch2b: secret "image-registry-tls" not found
Oct 30 22:06:51.124480 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:51.122211    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls podName:e297483a-0b86-4481-9597-0c89e9a6a7ac nodeName:}" failed. No retries permitted until 2026-10-30 22:06:59.122200857 +0000 UTC m=+45.692294147 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls") pod "image-registry-54b978dcc8-sch2b" (UID: "e297483a-0b86-4481-9597-0c89e9a6a7ac") : secret "image-registry-tls" not found
Oct 30 22:06:51.147117 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:51.147086    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:51.219094 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:51.218619    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9xdrk" event={"ID":"24c987a0-1cbf-4264-ba0e-586b738d08da","Type":"ContainerStarted","Data":"31f772fe2a6d58fe70488e3a111da2e196359fcb3315e0358a87bcdc36fdc99e"}
Oct 30 22:06:51.220447 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:51.220427    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/1.log"
Oct 30 22:06:51.220773 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:51.220755    2580 scope.go:122] "RemoveContainer" containerID="d60d0383fb24f739b8698d09dfc099d55168ef8507ac722ceb2547384d90c615"
Oct 30 22:06:51.220980 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:51.220957    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qbz82_openshift-console-operator(ea9c808c-62b4-4dc7-9761-bf6ee3700e34)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82" podUID="ea9c808c-62b4-4dc7-9761-bf6ee3700e34"
Oct 30 22:06:51.222098 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:51.222081    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:51.222170 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:51.222120    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea4b0705-2096-41da-b78b-b62590f236e8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-n42rm\" (UID: \"ea4b0705-2096-41da-b78b-b62590f236e8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-n42rm"
Oct 30 22:06:51.222263 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:51.222250    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:06:51.222310 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:51.222253    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:06:51.222310 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:51.222277    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b9d7c7dff-82x2r: secret "image-registry-tls" not found
Oct 30 22:06:51.222388 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:51.222311    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea4b0705-2096-41da-b78b-b62590f236e8-networking-console-plugin-cert podName:ea4b0705-2096-41da-b78b-b62590f236e8 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:59.222294346 +0000 UTC m=+45.792387633 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ea4b0705-2096-41da-b78b-b62590f236e8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-n42rm" (UID: "ea4b0705-2096-41da-b78b-b62590f236e8") : secret "networking-console-plugin-cert" not found
Oct 30 22:06:51.222388 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:51.222351    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls podName:b6651a05-7ba7-484f-8490-f22665bcb84f nodeName:}" failed. No retries permitted until 2026-10-30 22:06:59.22234019 +0000 UTC m=+45.792433470 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls") pod "image-registry-6b9d7c7dff-82x2r" (UID: "b6651a05-7ba7-484f-8490-f22665bcb84f") : secret "image-registry-tls" not found
Oct 30 22:06:51.222465 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:51.222408    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vqtw4\" (UID: \"fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"
Oct 30 22:06:51.222519 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:51.222507    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:06:51.222561 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:51.222542    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls podName:fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a nodeName:}" failed. No retries permitted until 2026-10-30 22:06:59.222533925 +0000 UTC m=+45.792627198 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vqtw4" (UID: "fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a") : secret "samples-operator-tls" not found
Oct 30 22:06:51.267670 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:51.267608    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9xdrk" podStartSLOduration=319.278002016 podStartE2EDuration="5m25.267591383s" totalImagesPullingTime="5.989589367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:51.26677511 +0000 UTC m=+37.836868409" watchObservedRunningTime="2026-10-30 22:06:51.267591383 +0000 UTC m=+37.837684678"
Oct 30 22:06:51.828393 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:51.828354    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:51.828616 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:51.828408    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/052e2399-f78d-4eb2-9da7-e2b098f86da7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:51.828616 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:51.828523    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:51.828616 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:51.828570    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 22:06:51.828616 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:51.828597    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls podName:052e2399-f78d-4eb2-9da7-e2b098f86da7 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:53.828576664 +0000 UTC m=+40.398669955 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mwjsl" (UID: "052e2399-f78d-4eb2-9da7-e2b098f86da7") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:51.828616 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:51.828618    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/052e2399-f78d-4eb2-9da7-e2b098f86da7-kube-rbac-proxy-cm podName:052e2399-f78d-4eb2-9da7-e2b098f86da7 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:53.828605758 +0000 UTC m=+40.398699045 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/052e2399-f78d-4eb2-9da7-e2b098f86da7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mwjsl" (UID: "052e2399-f78d-4eb2-9da7-e2b098f86da7") : configmap "kube-rbac-proxy" not found
Oct 30 22:06:51.844654 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:51.844621    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:52.225153 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:52.225061    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zcl2w" event={"ID":"d303600a-2390-4ba9-8ea8-a58bd505a109","Type":"ContainerStarted","Data":"02483b7024fe81466526736453566242e39ef6afefcb35c21cdfb1475f639f65"}
Oct 30 22:06:52.273921 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:52.273872    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zcl2w" podStartSLOduration=36.443697641 podStartE2EDuration="39.273857338s" totalImagesPullingTime="2.830159697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:52.273418726 +0000 UTC m=+38.843512021" watchObservedRunningTime="2026-10-30 22:06:52.273857338 +0000 UTC m=+38.843950654"
Oct 30 22:06:53.229134 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:53.229106    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:53.741596 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:53.741562    2580 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:06:53.741793 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:53.741628    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:06:53.742015 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:53.742001    2580 scope.go:122] "RemoveContainer" containerID="d60d0383fb24f739b8698d09dfc099d55168ef8507ac722ceb2547384d90c615"
Oct 30 22:06:53.742289 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:53.742253    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qbz82_openshift-console-operator(ea9c808c-62b4-4dc7-9761-bf6ee3700e34)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82" podUID="ea9c808c-62b4-4dc7-9761-bf6ee3700e34"
Oct 30 22:06:53.846157 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:53.846118    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:53.846157 ip-10-0-142-185 kubenswrapper[2580]: I1030 22:06:53.846160    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/052e2399-f78d-4eb2-9da7-e2b098f86da7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:53.846417 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:53.846291    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:53.846417 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:53.846332    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 22:06:53.846417 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:53.846360    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls podName:052e2399-f78d-4eb2-9da7-e2b098f86da7 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:57.846345696 +0000 UTC m=+44.416438985 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mwjsl" (UID: "052e2399-f78d-4eb2-9da7-e2b098f86da7") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:53.846417 ip-10-0-142-185 kubenswrapper[2580]: E1030 22:06:53.846399    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/052e2399-f78d-4eb2-9da7-e2b098f86da7-kube-rbac-proxy-cm podName:052e2399-f78d-4eb2-9da7-e2b098f86da7 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:57.846380672 +0000 UTC m=+44.416473961 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/052e2399-f78d-4eb2-9da7-e2b098f86da7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mwjsl" (UID: "052e2399-f78d-4eb2-9da7-e2b098f86da7") : configmap "kube-rbac-proxy" not found
Oct 30 22:06:54.588859 ip-10-0-142-185 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 22:06:54.597933 ip-10-0-142-185 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 22:06:54.598109 ip-10-0-142-185 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 22:06:54.598159 ip-10-0-142-185 systemd[1]: kubelet.service: Consumed 3.052s CPU time.
Oct 30 22:06:54.610783 ip-10-0-142-185 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 22:06:54.688939 ip-10-0-142-185 kubenswrapper[4889]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:06:54.688939 ip-10-0-142-185 kubenswrapper[4889]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 22:06:54.688939 ip-10-0-142-185 kubenswrapper[4889]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:06:54.688939 ip-10-0-142-185 kubenswrapper[4889]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:06:54.691005 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.690917    4889 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:54.691005 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691006    4889 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:54.691078 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691010    4889 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:54.691078 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691013    4889 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:54.691078 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691016    4889 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:54.691078 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691019    4889 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:54.691078 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691021    4889 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:54.691078 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691024    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:54.691078 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691027    4889 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:54.691078 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691029    4889 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:54.691078 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691032    4889 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:54.691078 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691035    4889 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:54.691078 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691037    4889 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:54.691078 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691040    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:54.691078 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691043    4889 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:54.691078 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691046    4889 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:54.691078 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691049    4889 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:54.691078 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691052    4889 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:54.691078 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691054    4889 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:54.691078 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691057    4889 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:54.691078 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691060    4889 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:54.691078 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691062    4889 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:54.691603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691073    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:54.691603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691076    4889 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:54.691603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691079    4889 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:54.691603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691082    4889 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:54.691603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691085    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:54.691603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691088    4889 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:54.691603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691091    4889 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:54.691603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691094    4889 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:54.691603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691097    4889 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:54.691603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691100    4889 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:54.691603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691102    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:54.691603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691105    4889 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:54.691603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691107    4889 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:54.691603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691110    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:54.691603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691113    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:54.691603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691116    4889 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:54.691603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691119    4889 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:54.691603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691123    4889 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:54.691603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691126    4889 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:54.692070 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691129    4889 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:54.692070 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691131    4889 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:54.692070 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691134    4889 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:54.692070 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691136    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:54.692070 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691139    4889 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:54.692070 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691141    4889 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:54.692070 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691144    4889 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:54.692070 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691146    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:54.692070 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691149    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:54.692070 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691152    4889 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:54.692070 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691155    4889 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:54.692070 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691157    4889 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:54.692070 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691159    4889 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:54.692070 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691162    4889 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:54.692070 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691167    4889 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:54.692070 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691169    4889 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:54.692070 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691172    4889 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:54.692070 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691174    4889 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:54.692070 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691177    4889 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:54.692070 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691180    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:54.692597 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691183    4889 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:54.692597 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691186    4889 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:54.692597 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691188    4889 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:54.692597 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691191    4889 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:54.692597 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691193    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:54.692597 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691196    4889 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:54.692597 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691201    4889 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:54.692597 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691205    4889 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:54.692597 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691207    4889 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:54.692597 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691210    4889 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:54.692597 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691213    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:54.692597 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691216    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:54.692597 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691218    4889 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:54.692597 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691239    4889 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:54.692597 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691242    4889 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:54.692597 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691245    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:54.692597 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691247    4889 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:54.692597 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691250    4889 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:54.692597 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691254    4889 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:54.693063 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691256    4889 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:54.693063 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691259    4889 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:54.693063 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691262    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:54.693063 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691265    4889 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:54.693063 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691268    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:54.693063 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691270    4889 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:54.693063 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691273    4889 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:54.693063 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691276    4889 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:54.693063 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691280    4889 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:54.693063 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691283    4889 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:54.693063 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691285    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:54.693063 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691288    4889 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:54.693063 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691291    4889 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:54.693063 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691294    4889 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:54.693063 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691296    4889 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:54.693063 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691299    4889 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:54.693063 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691301    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:54.693063 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691304    4889 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:54.693063 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691306    4889 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:54.693063 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691309    4889 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:54.693603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691311    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:54.693603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691314    4889 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:54.693603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691316    4889 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:54.693603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691319    4889 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:54.693603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691322    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:54.693603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.691324    4889 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:54.693603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692119    4889 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:54.693603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692127    4889 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:54.693603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692130    4889 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:54.693603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692134    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:54.693603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692136    4889 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:54.693603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692139    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:54.693603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692141    4889 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:54.693603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692144    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:54.693603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692147    4889 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:54.693603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692150    4889 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:54.693603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692152    4889 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:54.693603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692156    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:54.693603 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692158    4889 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:54.694082 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692161    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:54.694082 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692163    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:54.694082 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692166    4889 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:54.694082 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692169    4889 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:54.694082 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692171    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:54.694082 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692174    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:54.694082 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692176    4889 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:54.694082 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692179    4889 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:54.694082 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692181    4889 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:54.694082 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692184    4889 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:54.694082 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692186    4889 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:54.694082 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692189    4889 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:54.694082 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692191    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:54.694082 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692194    4889 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:54.694082 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692196    4889 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:54.694082 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692199    4889 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:54.694082 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692202    4889 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:54.694082 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692205    4889 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:54.694082 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692208    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:54.694585 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692210    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:54.694585 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692213    4889 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:54.694585 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692216    4889 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:54.694585 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692235    4889 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:54.694585 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692239    4889 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:54.694585 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692243    4889 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:54.694585 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692247    4889 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:54.694585 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692251    4889 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:54.694585 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692253    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:54.694585 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692256    4889 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:54.694585 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692259    4889 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:54.694585 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692262    4889 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:54.694585 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692264    4889 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:54.694585 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692267    4889 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:54.694585 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692269    4889 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:54.694585 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692274    4889 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:54.694585 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692279    4889 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:54.694585 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692282    4889 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:54.694585 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692285    4889 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:54.694585 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692288    4889 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:54.695106 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692291    4889 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:54.695106 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692294    4889 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:54.695106 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692297    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:54.695106 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692299    4889 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:54.695106 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692302    4889 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:54.695106 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692304    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:54.695106 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692307    4889 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:54.695106 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692309    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:54.695106 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692313    4889 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:54.695106 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692316    4889 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:54.695106 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692318    4889 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:54.695106 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692323    4889 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:54.695106 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692327    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:54.695106 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692330    4889 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:54.695106 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692333    4889 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:54.695106 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692335    4889 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:54.695106 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692338    4889 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:54.695106 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692341    4889 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:54.695106 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692343    4889 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:54.695596 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692346    4889 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:54.695596 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692349    4889 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:54.695596 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692351    4889 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:54.695596 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692354    4889 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:54.695596 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692356    4889 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:54.695596 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692359    4889 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:54.695596 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692361    4889 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:54.695596 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692364    4889 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:54.695596 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692366    4889 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:54.695596 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692370    4889 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:54.695596 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692373    4889 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:54.695596 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692376    4889 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:54.695596 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692378    4889 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:54.695596 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692381    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:54.695596 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692383    4889 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:54.695596 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692385    4889 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:54.695596 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692388    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:54.695596 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692391    4889 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:54.695596 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692393    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:54.695596 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692396    4889 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:54.696114 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692398    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:54.696114 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692401    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:54.696114 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692404    4889 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:54.696114 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692407    4889 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:54.696114 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692409    4889 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:54.696114 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692412    4889 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:54.696114 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692415    4889 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:54.696114 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692417    4889 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:54.696114 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692420    4889 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:54.696114 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692423    4889 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:54.696114 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692426    4889 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:54.696114 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692429    4889 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:54.696114 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692431    4889 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:54.696114 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692434    4889 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:54.696114 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692436    4889 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:54.696114 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692942    4889 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:54.696114 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692947    4889 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:54.696114 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692950    4889 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:54.696114 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692953    4889 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:54.696114 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692956    4889 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:54.696627 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692959    4889 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:54.696627 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692962    4889 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:54.696627 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692966    4889 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:54.696627 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692969    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:54.696627 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692971    4889 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:54.696627 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692974    4889 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:54.696627 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692976    4889 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:54.696627 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692979    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:54.696627 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692982    4889 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:54.696627 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692984    4889 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:54.696627 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692986    4889 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:54.696627 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692989    4889 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:54.696627 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692992    4889 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:54.696627 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692994    4889 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:54.696627 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.692997    4889 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:54.696627 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693011    4889 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:54.696627 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693015    4889 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:54.696627 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693019    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:54.696627 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693022    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:54.697142 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693026    4889 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:54.697142 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693030    4889 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:54.697142 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693033    4889 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:54.697142 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693036    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:54.697142 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693039    4889 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:54.697142 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693041    4889 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:54.697142 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693044    4889 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:54.697142 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693047    4889 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:54.697142 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693050    4889 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:54.697142 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693052    4889 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:54.697142 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693055    4889 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:54.697142 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693057    4889 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:54.697142 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693060    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:54.697142 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693062    4889 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:54.697142 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693066    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:54.697142 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693068    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:54.697142 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693072    4889 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:54.697142 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693076    4889 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:54.697142 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693080    4889 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:54.697682 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693083    4889 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:54.697682 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693086    4889 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:54.697682 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693088    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:54.697682 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693091    4889 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:54.697682 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693094    4889 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:54.697682 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693097    4889 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:54.697682 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693100    4889 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:54.697682 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693102    4889 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:54.697682 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693105    4889 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:54.697682 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693107    4889 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:54.697682 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693110    4889 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:54.697682 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693112    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:54.697682 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693115    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:54.697682 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693117    4889 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:54.697682 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693120    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:54.697682 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693122    4889 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:54.697682 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693125    4889 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:54.697682 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693128    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:54.697682 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693130    4889 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:54.698154 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693133    4889 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:54.698154 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693135    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:54.698154 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693138    4889 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:54.698154 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693140    4889 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:54.698154 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693142    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:54.698154 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693145    4889 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:54.698154 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693148    4889 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:54.698154 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693150    4889 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:54.698154 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693153    4889 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:54.698154 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693155    4889 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:54.698154 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693157    4889 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:54.698154 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693160    4889 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:54.698154 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693163    4889 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:54.698154 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693165    4889 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:54.698154 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693168    4889 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:54.698154 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693171    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:54.698154 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693173    4889 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:54.698154 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693176    4889 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:54.698154 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693178    4889 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:54.698154 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693181    4889 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:54.698658 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693185    4889 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:54.698658 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693187    4889 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:54.698658 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693190    4889 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:54.698658 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693192    4889 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:54.698658 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693195    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:54.698658 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693197    4889 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:54.698658 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693200    4889 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:54.698658 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693202    4889 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:54.698658 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693205    4889 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:54.698658 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693208    4889 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:54.698658 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693210    4889 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:54.698658 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693213    4889 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:54.698658 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693215    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:54.698658 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693218    4889 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:54.698658 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693237    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:54.698658 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693240    4889 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:54.698658 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693243    4889 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:54.698658 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693252    4889 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:54.698658 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693255    4889 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:54.698658 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693258    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:54.699191 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693261    4889 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:54.699191 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693263    4889 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:54.699191 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693266    4889 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:54.699191 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.693269    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:54.699191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693366    4889 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 22:06:54.699191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693374    4889 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 22:06:54.699191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693381    4889 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 22:06:54.699191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693386    4889 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 22:06:54.699191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693391    4889 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 22:06:54.699191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693394    4889 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 22:06:54.699191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693399    4889 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 22:06:54.699191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693403    4889 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 22:06:54.699191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693406    4889 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 22:06:54.699191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693411    4889 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 22:06:54.699191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693417    4889 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 22:06:54.699191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693422    4889 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 22:06:54.699191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693425    4889 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 22:06:54.699191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693428    4889 flags.go:64] FLAG: --cgroup-root=""
Oct 30 22:06:54.699191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693431    4889 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 22:06:54.699191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693434    4889 flags.go:64] FLAG: --client-ca-file=""
Oct 30 22:06:54.699191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693436    4889 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 22:06:54.699191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693439    4889 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 22:06:54.699191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693444    4889 flags.go:64] FLAG: --cluster-domain=""
Oct 30 22:06:54.699191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693447    4889 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 22:06:54.699804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693450    4889 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 22:06:54.699804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693453    4889 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 22:06:54.699804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693456    4889 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 22:06:54.699804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693460    4889 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 22:06:54.699804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693463    4889 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 22:06:54.699804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693468    4889 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 22:06:54.699804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693475    4889 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 22:06:54.699804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693478    4889 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 22:06:54.699804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693481    4889 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 22:06:54.699804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693484    4889 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 22:06:54.699804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693487    4889 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 22:06:54.699804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693490    4889 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 22:06:54.699804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693495    4889 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 22:06:54.699804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693498    4889 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 22:06:54.699804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693501    4889 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 22:06:54.699804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693504    4889 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 22:06:54.699804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693507    4889 flags.go:64] FLAG: --enable-server="true"
Oct 30 22:06:54.699804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693510    4889 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 22:06:54.699804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693515    4889 flags.go:64] FLAG: --event-burst="100"
Oct 30 22:06:54.699804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693518    4889 flags.go:64] FLAG: --event-qps="50"
Oct 30 22:06:54.699804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693521    4889 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 22:06:54.699804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693524    4889 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 22:06:54.699804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693527    4889 flags.go:64] FLAG: --eviction-hard=""
Oct 30 22:06:54.699804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693531    4889 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 22:06:54.700431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693534    4889 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 22:06:54.700431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693538    4889 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 22:06:54.700431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693541    4889 flags.go:64] FLAG: --eviction-soft=""
Oct 30 22:06:54.700431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693544    4889 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 22:06:54.700431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693547    4889 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 22:06:54.700431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693550    4889 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 22:06:54.700431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693553    4889 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 22:06:54.700431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693556    4889 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 22:06:54.700431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693559    4889 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 22:06:54.700431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693562    4889 flags.go:64] FLAG: --feature-gates=""
Oct 30 22:06:54.700431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693566    4889 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 22:06:54.700431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693569    4889 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 22:06:54.700431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693572    4889 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 22:06:54.700431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693576    4889 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 22:06:54.700431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693580    4889 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 22:06:54.700431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693583    4889 flags.go:64] FLAG: --help="false"
Oct 30 22:06:54.700431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693587    4889 flags.go:64] FLAG: --hostname-override="ip-10-0-142-185.ec2.internal"
Oct 30 22:06:54.700431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693590    4889 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 22:06:54.700431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693593    4889 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 22:06:54.700431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693596    4889 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 22:06:54.700431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693599    4889 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 22:06:54.700431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693603    4889 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 22:06:54.700431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693606    4889 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 22:06:54.700431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693609    4889 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 22:06:54.701026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693612    4889 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 22:06:54.701026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693615    4889 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 22:06:54.701026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693618    4889 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 22:06:54.701026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693621    4889 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 22:06:54.701026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693625    4889 flags.go:64] FLAG: --kube-reserved=""
Oct 30 22:06:54.701026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693627    4889 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 22:06:54.701026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693630    4889 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 22:06:54.701026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693633    4889 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 22:06:54.701026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693636    4889 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 22:06:54.701026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693639    4889 flags.go:64] FLAG: --lock-file=""
Oct 30 22:06:54.701026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693641    4889 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 22:06:54.701026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693644    4889 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 22:06:54.701026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693647    4889 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 22:06:54.701026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693653    4889 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 22:06:54.701026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693656    4889 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 22:06:54.701026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693659    4889 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 22:06:54.701026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693662    4889 flags.go:64] FLAG: --logging-format="text"
Oct 30 22:06:54.701026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693665    4889 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 22:06:54.701026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693668    4889 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 22:06:54.701026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693671    4889 flags.go:64] FLAG: --manifest-url=""
Oct 30 22:06:54.701026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693674    4889 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 22:06:54.701026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693684    4889 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 22:06:54.701026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693689    4889 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 22:06:54.701026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693692    4889 flags.go:64] FLAG: --max-pods="110"
Oct 30 22:06:54.701026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693696    4889 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 22:06:54.701663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693698    4889 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 22:06:54.701663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693701    4889 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 22:06:54.701663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693704    4889 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 22:06:54.701663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693707    4889 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 22:06:54.701663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693710    4889 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 22:06:54.701663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693713    4889 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 22:06:54.701663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693720    4889 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 22:06:54.701663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693724    4889 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 22:06:54.701663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693727    4889 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 22:06:54.701663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693730    4889 flags.go:64] FLAG: --pod-cidr=""
Oct 30 22:06:54.701663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693733    4889 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 22:06:54.701663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693735    4889 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 22:06:54.701663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693738    4889 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 22:06:54.701663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693741    4889 flags.go:64] FLAG: --port="10250"
Oct 30 22:06:54.701663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693744    4889 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 22:06:54.701663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693747    4889 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0585283188777a4e7"
Oct 30 22:06:54.701663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693750    4889 flags.go:64] FLAG: --qos-reserved=""
Oct 30 22:06:54.701663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693753    4889 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 22:06:54.701663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693756    4889 flags.go:64] FLAG: --register-node="true"
Oct 30 22:06:54.701663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693759    4889 flags.go:64] FLAG: --register-with-taints=""
Oct 30 22:06:54.701663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693763    4889 flags.go:64] FLAG: --registry-burst="10"
Oct 30 22:06:54.701663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693766    4889 flags.go:64] FLAG: --registry-qps="5"
Oct 30 22:06:54.701663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693768    4889 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 22:06:54.701663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693771    4889 flags.go:64] FLAG: --reserved-memory=""
Oct 30 22:06:54.701663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693775    4889 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 22:06:54.701663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693778    4889 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 22:06:54.702329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693781    4889 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 22:06:54.702329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693784    4889 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 22:06:54.702329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693788    4889 flags.go:64] FLAG: --runonce="false"
Oct 30 22:06:54.702329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693792    4889 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 22:06:54.702329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693795    4889 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 22:06:54.702329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693798    4889 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 22:06:54.702329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693801    4889 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 22:06:54.702329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693804    4889 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 22:06:54.702329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693806    4889 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 22:06:54.702329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693809    4889 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 22:06:54.702329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693813    4889 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 22:06:54.702329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693816    4889 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 22:06:54.702329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693819    4889 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 22:06:54.702329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693822    4889 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 22:06:54.702329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693825    4889 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 22:06:54.702329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693828    4889 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 22:06:54.702329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693831    4889 flags.go:64] FLAG: --system-cgroups=""
Oct 30 22:06:54.702329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693834    4889 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 22:06:54.702329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693839    4889 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 22:06:54.702329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693842    4889 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 22:06:54.702329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693845    4889 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 22:06:54.702329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693849    4889 flags.go:64] FLAG: --tls-min-version=""
Oct 30 22:06:54.702329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693852    4889 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 22:06:54.702329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693855    4889 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 22:06:54.702329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693857    4889 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 22:06:54.703297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693860    4889 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 22:06:54.703297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693863    4889 flags.go:64] FLAG: --v="2"
Oct 30 22:06:54.703297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693867    4889 flags.go:64] FLAG: --version="false"
Oct 30 22:06:54.703297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693871    4889 flags.go:64] FLAG: --vmodule=""
Oct 30 22:06:54.703297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693875    4889 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 22:06:54.703297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.693878    4889 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 22:06:54.703297 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694014    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:54.703297 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694018    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:54.703297 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694020    4889 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:54.703297 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694025    4889 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:54.703297 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694029    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:54.703297 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694031    4889 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:54.703297 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694034    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:54.703297 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694037    4889 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:54.703297 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694039    4889 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:54.703297 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694042    4889 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:54.703297 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694045    4889 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:54.703297 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694048    4889 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:54.703297 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694050    4889 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:54.703297 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694053    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:54.703297 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694056    4889 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:54.703986 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694059    4889 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:54.703986 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694062    4889 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:54.703986 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694064    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:54.703986 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694067    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:54.703986 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694069    4889 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:54.703986 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694072    4889 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:54.703986 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694075    4889 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:54.703986 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694077    4889 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:54.703986 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694080    4889 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:54.703986 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694083    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:54.703986 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694085    4889 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:54.703986 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694088    4889 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:54.703986 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694091    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:54.703986 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694094    4889 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:54.703986 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694096    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:54.703986 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694099    4889 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:54.703986 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694101    4889 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:54.703986 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694104    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:54.703986 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694107    4889 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:54.704629 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694109    4889 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:54.704629 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694113    4889 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:54.704629 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694116    4889 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:54.704629 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694119    4889 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:54.704629 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694122    4889 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:54.704629 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694124    4889 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:54.704629 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694127    4889 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:54.704629 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694130    4889 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:54.704629 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694132    4889 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:54.704629 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694135    4889 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:54.704629 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694138    4889 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:54.704629 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694140    4889 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:54.704629 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694143    4889 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:54.704629 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694145    4889 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:54.704629 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694148    4889 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:54.704629 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694151    4889 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:54.704629 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694154    4889 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:54.704629 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694156    4889 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:54.704629 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694158    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:54.704629 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694161    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:54.705110 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694163    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:54.705110 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694166    4889 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:54.705110 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694168    4889 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:54.705110 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694171    4889 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:54.705110 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694173    4889 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:54.705110 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694176    4889 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:54.705110 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694178    4889 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:54.705110 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694181    4889 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:54.705110 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694183    4889 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:54.705110 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694186    4889 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:54.705110 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694189    4889 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:54.705110 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694191    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:54.705110 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694194    4889 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:54.705110 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694197    4889 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:54.705110 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694201    4889 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:54.705110 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694205    4889 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:54.705110 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694209    4889 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:54.705110 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694212    4889 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:54.705110 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694216    4889 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:54.705628 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694235    4889 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:54.705628 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694240    4889 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:54.705628 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694244    4889 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:54.705628 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694249    4889 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:54.705628 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694253    4889 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:54.705628 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694257    4889 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:54.705628 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694260    4889 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:54.705628 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694262    4889 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:54.705628 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694266    4889 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:54.705628 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694268    4889 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:54.705628 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694271    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:54.705628 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694274    4889 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:54.705628 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694276    4889 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:54.705628 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694279    4889 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:54.705628 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694281    4889 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:54.705628 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694284    4889 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:54.705628 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694286    4889 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:54.705628 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694289    4889 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:54.705628 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694291    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:54.706434 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694294    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:54.706434 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694296    4889 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:54.706434 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694299    4889 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:54.706434 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694302    4889 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:54.706434 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694304    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:54.706434 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694307    4889 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:54.706434 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694309    4889 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:54.706434 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694313    4889 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:54.706434 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694318    4889 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:54.706434 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694321    4889 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:54.706434 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694323    4889 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:54.706434 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694326    4889 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:54.706434 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694328    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:54.706434 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694331    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:54.706434 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.694358    4889 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:06:54.706434 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694394    4889 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:54.706884 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694399    4889 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:54.706884 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694401    4889 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:54.706884 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694404    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:54.706884 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694407    4889 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:54.706884 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694410    4889 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:54.706884 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694412    4889 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:54.706884 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694416    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:54.706884 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694418    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:54.706884 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694421    4889 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:54.706884 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694424    4889 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:54.706884 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694426    4889 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:54.706884 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694429    4889 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:54.706884 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694432    4889 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:54.706884 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694434    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:54.706884 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694437    4889 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:54.706884 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694440    4889 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:54.706884 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694442    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:54.706884 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694445    4889 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:54.706884 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694447    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:54.706884 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694450    4889 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:54.707453 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694453    4889 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:54.707453 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694455    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:54.707453 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694458    4889 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:54.707453 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694463    4889 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:54.707453 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694466    4889 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:54.707453 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694469    4889 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:54.707453 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694472    4889 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:54.707453 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694475    4889 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:54.707453 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694478    4889 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:54.707453 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694480    4889 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:54.707453 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694483    4889 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:54.707453 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694486    4889 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:54.707453 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694488    4889 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:54.707453 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694491    4889 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:54.707453 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694493    4889 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:54.707453 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694496    4889 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:54.707453 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694498    4889 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:54.707453 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694501    4889 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:54.707453 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694504    4889 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:54.707453 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694507    4889 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:54.708025 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694509    4889 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:54.708025 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694512    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:54.708025 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694514    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:54.708025 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694517    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:54.708025 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694519    4889 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:54.708025 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694522    4889 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:54.708025 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694524    4889 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:54.708025 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694527    4889 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:54.708025 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694529    4889 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:54.708025 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694531    4889 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:54.708025 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694534    4889 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:54.708025 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694536    4889 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:54.708025 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694539    4889 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:54.708025 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694553    4889 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:54.708025 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694556    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:54.708025 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694559    4889 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:54.708025 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694562    4889 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:54.708025 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694566    4889 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:54.708025 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694570    4889 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:54.708672 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694574    4889 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:54.708672 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694577    4889 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:54.708672 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694580    4889 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:54.708672 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694582    4889 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:54.708672 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694585    4889 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:54.708672 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694589    4889 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:54.708672 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694593    4889 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:54.708672 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694596    4889 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:54.708672 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694599    4889 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:54.708672 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694601    4889 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:54.708672 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694604    4889 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:54.708672 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694606    4889 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:54.708672 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694609    4889 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:54.708672 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694612    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:54.708672 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694615    4889 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:54.708672 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694618    4889 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:54.708672 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694620    4889 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:54.708672 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694623    4889 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:54.708672 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694625    4889 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:54.709151 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694628    4889 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:54.709151 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694630    4889 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:54.709151 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694633    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:54.709151 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694635    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:54.709151 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694638    4889 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:54.709151 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694641    4889 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:54.709151 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694643    4889 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:54.709151 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694646    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:54.709151 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694648    4889 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:54.709151 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694652    4889 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:54.709151 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694654    4889 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:54.709151 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694658    4889 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:54.709151 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694661    4889 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:54.709151 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694663    4889 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:54.709151 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694666    4889 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:54.709151 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694668    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:54.709151 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694671    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:54.709151 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694673    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:54.709151 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694676    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:54.709640 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694678    4889 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:54.709640 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694681    4889 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:54.709640 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694683    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:54.709640 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694686    4889 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:54.709640 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694689    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:54.709640 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694691    4889 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:54.709640 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694694    4889 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:54.709640 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.694697    4889 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:54.709640 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.699015    4889 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 22:06:54.709640 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.699029    4889 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 22:06:54.709640 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.699046    4889 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 22:06:54.709640 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.699051    4889 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 22:06:54.709640 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699123    4889 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:54.709640 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699129    4889 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:54.709640 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699132    4889 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:54.709640 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699135    4889 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:54.709640 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699139    4889 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:54.709640 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699142    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:54.709640 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699146    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:54.710102 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699150    4889 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:54.710102 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699154    4889 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:54.710102 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699157    4889 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:54.710102 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699160    4889 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:54.710102 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699163    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:54.710102 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699167    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:54.710102 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699175    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:54.710102 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699178    4889 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:54.710102 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699182    4889 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:54.710102 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699185    4889 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:54.710102 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699189    4889 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:54.710102 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699191    4889 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:54.710102 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699194    4889 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:54.710102 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699197    4889 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:54.710102 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699199    4889 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:54.710102 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699202    4889 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:54.710102 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699205    4889 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:54.710102 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699207    4889 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:54.710102 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699210    4889 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:54.710664 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699213    4889 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:54.710664 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699215    4889 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:54.710664 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699218    4889 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:54.710664 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699235    4889 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:54.710664 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699238    4889 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:54.710664 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699241    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:54.710664 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699243    4889 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:54.710664 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699246    4889 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:54.710664 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699248    4889 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:54.710664 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699251    4889 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:54.710664 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699254    4889 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:54.710664 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699256    4889 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:54.710664 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699259    4889 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:54.710664 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699261    4889 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:54.710664 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699263    4889 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:54.710664 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699266    4889 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:54.710664 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699268    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:54.710664 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699271    4889 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:54.710664 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699274    4889 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:54.710664 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699276    4889 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:54.711181 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699284    4889 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:54.711181 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699287    4889 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:54.711181 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699290    4889 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:54.711181 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699292    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:54.711181 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699295    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:54.711181 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699297    4889 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:54.711181 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699300    4889 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:54.711181 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699302    4889 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:54.711181 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699305    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:54.711181 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699307    4889 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:54.711181 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699310    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:54.711181 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699313    4889 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:54.711181 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699315    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:54.711181 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699318    4889 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:54.711181 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699321    4889 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:54.711181 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699323    4889 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:54.711181 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699325    4889 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:54.711181 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699328    4889 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:54.711181 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699330    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:54.711657 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699333    4889 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:54.711657 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699336    4889 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:54.711657 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699338    4889 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:54.711657 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699341    4889 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:54.711657 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699343    4889 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:54.711657 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699346    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:54.711657 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699349    4889 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:54.711657 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699351    4889 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:54.711657 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699354    4889 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:54.711657 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699356    4889 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:54.711657 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699359    4889 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:54.711657 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699361    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:54.711657 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699364    4889 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:54.711657 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699367    4889 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:54.711657 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699376    4889 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:54.711657 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699379    4889 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:54.711657 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699382    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:54.711657 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699385    4889 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:54.711657 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699387    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:54.712129 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699390    4889 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:54.712129 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699392    4889 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:54.712129 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699395    4889 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:54.712129 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699397    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:54.712129 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699400    4889 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:54.712129 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699402    4889 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:54.712129 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699405    4889 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:54.712129 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699408    4889 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:54.712129 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699410    4889 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:54.712129 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699413    4889 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:54.712129 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699416    4889 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:54.712129 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699418    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:54.712129 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699421    4889 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:54.712129 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699423    4889 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:54.712129 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699425    4889 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:54.712129 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699428    4889 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:54.712129 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699431    4889 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:54.712129 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699439    4889 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:54.712129 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699442    4889 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:54.712609 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699445    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:54.712609 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699448    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:54.712609 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699450    4889 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:54.712609 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.699478    4889 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:06:54.712609 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699677    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:54.712609 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699681    4889 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:54.712609 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699684    4889 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:54.712609 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699687    4889 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:54.712609 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699689    4889 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:54.712609 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699698    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:54.712609 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699700    4889 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:54.712609 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699703    4889 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:54.712609 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699705    4889 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:54.712609 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699708    4889 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:54.712609 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699710    4889 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:54.712609 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699713    4889 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:54.712997 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699715    4889 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:54.712997 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699718    4889 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:54.712997 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699720    4889 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:54.712997 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699723    4889 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:54.712997 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699725    4889 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:54.712997 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699728    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:54.712997 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699730    4889 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:54.712997 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699732    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:54.712997 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699735    4889 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:54.712997 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699739    4889 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:54.712997 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699742    4889 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:54.712997 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699745    4889 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:54.712997 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699749    4889 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:54.712997 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699753    4889 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:54.712997 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699756    4889 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:54.712997 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699759    4889 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:54.712997 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699762    4889 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:54.712997 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699765    4889 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:54.712997 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699767    4889 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:54.713469 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699769    4889 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:54.713469 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699772    4889 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:54.713469 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699774    4889 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:54.713469 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699777    4889 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:54.713469 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699779    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:54.713469 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699781    4889 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:54.713469 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699784    4889 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:54.713469 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699792    4889 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:54.713469 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699795    4889 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:54.713469 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699797    4889 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:54.713469 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699800    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:54.713469 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699802    4889 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:54.713469 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699805    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:54.713469 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699808    4889 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:54.713469 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699810    4889 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:54.713469 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699813    4889 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:54.713469 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699816    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:54.713469 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699818    4889 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:54.713469 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699821    4889 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:54.713469 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699824    4889 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:54.713928 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699826    4889 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:54.713928 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699828    4889 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:54.713928 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699831    4889 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:54.713928 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699834    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:54.713928 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699836    4889 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:54.713928 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699839    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:54.713928 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699841    4889 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:54.713928 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699844    4889 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:54.713928 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699846    4889 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:54.713928 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699848    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:54.713928 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699851    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:54.713928 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699853    4889 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:54.713928 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699856    4889 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:54.713928 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699859    4889 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:54.713928 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699861    4889 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:54.713928 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699863    4889 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:54.713928 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699866    4889 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:54.713928 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699868    4889 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:54.713928 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699871    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:54.713928 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699873    4889 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:54.714430 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699881    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:54.714430 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699884    4889 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:54.714430 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699886    4889 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:54.714430 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699889    4889 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:54.714430 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699891    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:54.714430 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699893    4889 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:54.714430 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699896    4889 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:54.714430 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699898    4889 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:54.714430 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699901    4889 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:54.714430 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699903    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:54.714430 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699906    4889 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:54.714430 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699908    4889 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:54.714430 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699911    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:54.714430 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699913    4889 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:54.714430 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699916    4889 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:54.714430 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699918    4889 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:54.714430 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699921    4889 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:54.714430 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699923    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:54.714430 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699925    4889 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:54.714905 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699928    4889 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:54.714905 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699930    4889 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:54.714905 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699933    4889 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:54.714905 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699935    4889 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:54.714905 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699937    4889 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:54.714905 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699940    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:54.714905 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699945    4889 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:54.714905 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699947    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:54.714905 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699950    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:54.714905 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699952    4889 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:54.714905 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699955    4889 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:54.714905 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699957    4889 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:54.714905 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699960    4889 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:54.714905 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699962    4889 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:54.714905 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699971    4889 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:54.714905 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.699974    4889 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:54.715471 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.699997    4889 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:06:54.715471 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700036    4889 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:54.715471 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700040    4889 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:54.715471 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700043    4889 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:54.715471 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700045    4889 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:54.715471 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700048    4889 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:54.715471 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700051    4889 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:54.715471 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700053    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:54.715471 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700057    4889 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:54.715471 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700060    4889 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:54.715471 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700062    4889 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:54.715471 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700065    4889 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:54.715471 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700068    4889 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:54.715471 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700070    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:54.715471 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700073    4889 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:54.715471 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700075    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:54.715877 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700078    4889 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:54.715877 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700081    4889 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:54.715877 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700084    4889 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:54.715877 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700086    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:54.715877 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700089    4889 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:54.715877 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700091    4889 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:54.715877 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700095    4889 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:54.715877 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700098    4889 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:54.715877 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700100    4889 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:54.715877 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700103    4889 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:54.715877 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700106    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:54.715877 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700108    4889 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:54.715877 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700111    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:54.715877 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700113    4889 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:54.715877 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700116    4889 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:54.715877 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700124    4889 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:54.715877 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700127    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:54.715877 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700129    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:54.715877 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700131    4889 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:54.715877 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700134    4889 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:54.716386 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700137    4889 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:54.716386 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700139    4889 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:54.716386 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700142    4889 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:54.716386 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700144    4889 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:54.716386 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700149    4889 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:54.716386 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700151    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:54.716386 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700154    4889 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:54.716386 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700156    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:54.716386 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700159    4889 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:54.716386 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700162    4889 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:54.716386 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700164    4889 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:54.716386 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700167    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:54.716386 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700169    4889 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:54.716386 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700172    4889 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:54.716386 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700174    4889 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:54.716386 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700177    4889 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:54.716386 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700180    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:54.716386 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700183    4889 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:54.716386 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700187    4889 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:54.716894 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700190    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:54.716894 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700192    4889 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:54.716894 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700195    4889 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:54.716894 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700197    4889 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:54.716894 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700200    4889 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:54.716894 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700202    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:54.716894 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700205    4889 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:54.716894 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700207    4889 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:54.716894 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700210    4889 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:54.716894 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700243    4889 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:54.716894 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700247    4889 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:54.716894 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700249    4889 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:54.716894 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700252    4889 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:54.716894 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700255    4889 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:54.716894 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700257    4889 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:54.716894 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700260    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:54.716894 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700263    4889 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:54.716894 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700267    4889 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:54.716894 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700270    4889 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:54.717372 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700272    4889 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:54.717372 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700275    4889 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:54.717372 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700278    4889 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:54.717372 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700280    4889 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:54.717372 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700283    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:54.717372 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700285    4889 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:54.717372 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700288    4889 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:54.717372 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700290    4889 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:54.717372 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700293    4889 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:54.717372 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700295    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:54.717372 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700298    4889 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:54.717372 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700300    4889 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:54.717372 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700304    4889 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:54.717372 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700306    4889 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:54.717372 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700309    4889 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:54.717372 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700312    4889 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:54.717372 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700314    4889 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:54.717372 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700317    4889 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:54.717372 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700319    4889 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:54.717372 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700322    4889 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:54.717851 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700324    4889 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:54.717851 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700327    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:54.717851 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700329    4889 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:54.717851 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700338    4889 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:54.717851 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700340    4889 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:54.717851 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700345    4889 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:54.717851 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700349    4889 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:54.717851 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700352    4889 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:54.717851 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700356    4889 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:54.717851 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700360    4889 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:54.717851 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700364    4889 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:54.717851 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700367    4889 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:54.717851 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:54.700371    4889 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:54.717851 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.700431    4889 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 22:06:54.717851 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.701465    4889 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 22:06:54.717851 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.701522    4889 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 22:06:54.717851 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.701718    4889 server.go:1014] "Starting client certificate rotation"
Oct 30 22:06:54.717851 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.701887    4889 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 22:06:54.718344 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.701957    4889 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 22:01:13 +0000 UTC" deadline="2028-07-03 12:41:47.659512576 +0000 UTC"
Oct 30 22:06:54.718344 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.702015    4889 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14678h34m52.95749939s"
Oct 30 22:06:54.718344 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.702297    4889 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 22:06:54.718344 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.704775    4889 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 22:06:54.718344 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.706033    4889 log.go:25] "Validated CRI v1 runtime API"
Oct 30 22:06:54.718344 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.707700    4889 log.go:25] "Validated CRI v1 image API"
Oct 30 22:06:54.718344 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.708482    4889 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 22:06:54.718344 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.710398    4889 fs.go:135] Filesystem UUIDs: map[2a35b543-9686-42b6-bb7d-21827bdd91c2:/dev/nvme0n1p3 615c480a-1685-45d6-885f-06ce32897ea8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 22:06:54.718552 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.710414    4889 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/03750ea778f9d46aafdb422942074845d557e3711abaffa235c54ae20950287d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/03750ea778f9d46aafdb422942074845d557e3711abaffa235c54ae20950287d/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/109c10c088184bcb0a693845404521cea84a57da050db42739d6f87e72638560/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/109c10c088184bcb0a693845404521cea84a57da050db42739d6f87e72638560/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/20d5b724c1ef9e807e1f97ce33a81114cbb8ebca9d46488a059e263ff1a7e1ee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/20d5b724c1ef9e807e1f97ce33a81114cbb8ebca9d46488a059e263ff1a7e1ee/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/293e7aaa41bae7e7f8f02cc4f11203729a2c7a3b847ff7e0b29819b5671d3a83/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/293e7aaa41bae7e7f8f02cc4f11203729a2c7a3b847ff7e0b29819b5671d3a83/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3681ee3cc46929a2fb642aa02d8f7701443039483d0c39517a43ace21b066d3f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3681ee3cc46929a2fb642aa02d8f7701443039483d0c39517a43ace21b066d3f/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3739156ea50346f67c1f1c70b6a7bd166cc518bbce04c34908a9c1148695a299/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3739156ea50346f67c1f1c70b6a7bd166cc518bbce04c34908a9c1148695a299/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3858b6612025b28ff98838f7ebf46418f61ea0e479908653b17a4d70c3bb3908/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3858b6612025b28ff98838f7ebf46418f61ea0e479908653b17a4d70c3bb3908/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/41bec762b666a037acb1960a7ddf13317cfcc5a451f2d3c7aab97a47dd7bd6e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/41bec762b666a037acb1960a7ddf13317cfcc5a451f2d3c7aab97a47dd7bd6e4/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/471d2921a86a8eaa2f2bb973796671cf840e2862e459183ac3d4afe68995188b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/471d2921a86a8eaa2f2bb973796671cf840e2862e459183ac3d4afe68995188b/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/473049b96ecb4925bb44684d88d7ed4a537e37099bdd407ebe98583432f67887/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/473049b96ecb4925bb44684d88d7ed4a537e37099bdd407ebe98583432f67887/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f2b127804d27f49d0c9bed28f086e3bb999bbd9fa415e695653470e9034f375/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f2b127804d27f49d0c9bed28f086e3bb999bbd9fa415e695653470e9034f375/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4faf0685059f939c00321305f975604de2c8a02d6a234aa7fb7bd4792e0661d2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4faf0685059f939c00321305f975604de2c8a02d6a234aa7fb7bd4792e0661d2/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/784af23f4ecb79948e4642e53991b673c4b33f50f0bfb91aeb5ef7a88ca9ab7d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/784af23f4ecb79948e4642e53991b673c4b33f50f0bfb91aeb5ef7a88ca9ab7d/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7d1343f6fe0cbba61605e7232037aa2743561125a07d7556dbdfd92daed9210c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7d1343f6fe0cbba61605e7232037aa2743561125a07d7556dbdfd92daed9210c/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/907de9fd04680f90043bccba46fefda4567401cf21b11ec468c4bc55d2492d6a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/907de9fd04680f90043bccba46fefda4567401cf21b11ec468c4bc55d2492d6a/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/940a21805282e57baa73037cb6e8c0073b44aa3db60ee4e376faa4a835331ae7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/940a21805282e57baa73037cb6e8c0073b44aa3db60ee4e376faa4a835331ae7/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aa6e8f0edb6759c894052cf03580233f00ba2d529c93f92e36a4b1b9efe9f64b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aa6e8f0edb6759c894052cf03580233f00ba2d529c93f92e36a4b1b9efe9f64b/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c543d14496ba1d58230b67dac2cad4b80d74f21e802d0bc59043dfd072cbcee4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c543d14496ba1d58230b67dac2cad4b80d74f21e802d0bc59043dfd072cbcee4/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da6d5bc0a53d2d71c94f4708be735cb287c3b7824e15690d1f3472fb87cb3453/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da6d5bc0a53d2d71c94f4708be735cb287c3b7824e15690d1f3472fb87cb3453/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e639ca449fd6481bd8538d41484639a47ad2d027e5d81ec2deb1b336f6c6c617/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e639ca449fd6481bd8538d41484639a47ad2d027e5d81ec2deb1b336f6c6c617/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eac8bfd37e67bf7de6667108361618f1e829410e6a381e7b137b3781e55f3ce4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eac8bfd37e67bf7de6667108361618f1e829410e6a381e7b137b3781e55f3ce4/userdata/shm major:0 minor:295 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f3a4ea6cd48a49887913384ad1cc06bef23dfe6263904754a20538780da8eea3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f3a4ea6cd48a49887913384ad1cc06bef23dfe6263904754a20538780da8eea3/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00b74f51-6c71-45f2-97af-6aea20b9eeaf/volumes/kubernetes.io~projected/kube-api-access-b66td:{mountpoint:/var/lib/kubelet/pods/00b74f51-6c71-45f2-97af-6aea20b9eeaf/volumes/kubernetes.io~projected/kube-api-access-b66td major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/052e2399-f78d-4eb2-9da7-e2b098f86da7/volumes/kubernetes.io~projected/kube-api-access-q5cps:{mountpoint:/var/lib/kubelet/pods/052e2399-f78d-4eb2-9da7-e2b098f86da7/volumes/kubernetes.io~projected/kube-api-access-q5cps major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10d1cecc-9e30-4354-8b42-60ddceadff44/volumes/kubernetes.io~projected/kube-api-access-b4wvm:{mountpoint:/var/lib/kubelet/pods/10d1cecc-9e30-4354-8b42-60ddceadff44/volumes/kubernetes.io~projected/kube-api-access-b4wvm major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e444015-5346-4106-93aa-ef43eb9d0be6/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/1e444015-5346-4106-93aa-ef43eb9d0be6/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/200052bd-2f0a-46d2-8b9b-6bd250b17b05/volumes/kubernetes.io~projected/kube-api-access-h6cjm:{mountpoint:/var/lib/kubelet/pods/200052bd-2f0a-46d2-8b9b-6bd250b17b05/volumes/kubernetes.io~projected/kube-api-access-h6cjm major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24c987a0-1cbf-4264-ba0e-586b738d08da/volumes/kubernetes.io~projected/kube-api-access-h824x:{mountpoint:/var/lib/kubelet/pods/24c987a0-1cbf-4264-ba0e-586b738d08da/volumes/kubernetes.io~projected/kube-api-access-h824x major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ec19290-25c4-4b52-8173-22c14e7c9056/volumes/kubernetes.io~projected/kube-api-access-5k4xc:{mountpoint:/var/lib/kubelet/pods/2ec19290-25c4-4b52-8173-22c14e7c9056/volumes/kubernetes.io~projected/kube-api-access-5k4xc major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f403b34-f155-49e6-b327-8ad89d00bbc3/volumes/kubernetes.io~projected/kube-api-access-qdqmz:{mountpoint:/var/lib/kubelet/pods/3f403b34-f155-49e6-b327-8ad89d00bbc3/volumes/kubernetes.io~projected/kube-api-access-qdqmz major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4049e53b-bffe-40fb-beef-22acc171315a/volumes/kubernetes.io~projected/kube-api-access-c7fw6:{mountpoint:/var/lib/kubelet/pods/4049e53b-bffe-40fb-beef-22acc171315a/volumes/kubernetes.io~projected/kube-api-access-c7fw6 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47dc8b20-b520-4f4d-bd48-74b78ba300ef/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/47dc8b20-b520-4f4d-bd48-74b78ba300ef/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47dc8b20-b520-4f4d-bd48-74b78ba300ef/volumes/kubernetes.io~projected/kube-api-access-swswj:{mountpoint:/var/lib/kubelet/pods/47dc8b20-b520-4f4d-bd48-74b78ba300ef/volumes/kubernetes.io~projected/kube-api-access-swswj major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47dc8b20-b520-4f4d-bd48-74b78ba300ef/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/47dc8b20-b520-4f4d-bd48-74b78ba300ef/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9/volumes/kubernetes.io~projected/kube-api-access-k8ltk:{mountpoint:/var/lib/kubelet/pods/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9/volumes/kubernetes.io~projected/kube-api-access-k8ltk major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b84639d-6b5d-4795-a356-8755ea234585/volumes/kubernetes.io~projected/kube-api-access-92nhs:{mountpoint:/var/lib/kubelet/pods/5b84639d-6b5d-4795-a356-8755ea234585/volumes/kubernetes.io~projected/kube-api-access-92nhs major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b84639d-6b5d-4795-a356-8755ea234585/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/5b84639d-6b5d-4795-a356-8755ea234585/volumes/kubernetes.io~secret/serving-cert major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/605c1b60-1fa5-4257-a588-4b2e0a78bcca/volumes/kubernetes.io~projected/kube-api-access-dr2ww:{mountpoint:/var/lib/kubelet/pods/605c1b60-1fa5-4257-a588-4b2e0a78bcca/volumes/kubernetes.io~projected/kube-api-access-dr2ww major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/605c1b60-1fa5-4257-a588-4b2e0a78bcca/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/605c1b60-1fa5-4257-a588-4b2e0a78bcca/volumes/kubernetes.io~secret/default-certificate major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/605c1b60-1fa5-4257-a588-4b2e0a78bcca/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/605c1b60-1fa5-4257-a588-4b2e0a78bcca/volumes/kubernetes.io~secret/stats-auth major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/801d7671-e004-4da3-898e-664eb2f1b5c3/volumes/kubernetes.io~projected/kube-api-access-jrsqc:{mountpoint:/var/lib/kubelet/pods/801d7671-e004-4da3-898e-664eb2f1b5c3/volumes/kubernetes.io~projected/kube-api-access-jrsqc major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88fe4dfd-dcda-444d-871b-24ca3fb10d77/volumes/kubernetes.io~projected/kube-api-access-xrm4r:{mountpoint:/var/lib/kubelet/pods/88fe4dfd-dcda-444d-871b-24ca3fb10d77/volumes/kubernetes.io~projected/kube-api-access-xrm4r major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/965e7b7f-9044-4c69-be4d-06ef9a1044e2/volumes/kubernetes.io~projected/kube-api-access-7qqhv:{mountpoint:/var/lib/kubelet/pods/965e7b7f-9044-4c69-be4d-06ef9a1044e2/volumes/kubernetes.io~projected/kube-api-access-7qqhv major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/965e7b7f-9044-4c69-be4d-06ef9a1044e2/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/965e7b7f-9044-4c69-be4d-06ef9a1044e2/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d1eecff-b22f-4864-a121-9fb507e783f2/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/9d1eecff-b22f-4864-a121-9fb507e783f2/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6651a05-7ba7-484f-8490-f22665bcb84f/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/b6651a05-7ba7-484f-8490-f22665bcb84f/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6651a05-7ba7-484f-8490-f22665bcb84f/volumes/kubernetes.io~projected/kube-api-access-vr8dq:{mountpoint:/var/lib/kubelet/pods/b6651a05-7ba7-484f-8490-f22665bcb84f/volumes/kubernetes.io~projected/kube-api-access-vr8dq major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6651a05-7ba7-484f-8490-f22665bcb84f/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/b6651a05-7ba7-484f-8490-f22665bcb84f/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6651a05-7ba7-484f-8490-f22665bcb84f/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/b6651a05-7ba7-484f-8490-f22665bcb84f/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8ee94be-1472-42e7-bf6e-d13e58671eeb/volumes/kubernetes.io~projected/kube-api-access-h7wm8:{mountpoint:/var/lib/kubelet/pods/b8ee94be-1472-42e7-bf6e-d13e58671eeb/volumes/kubernetes.io~projected/kube-api-access-h7wm8 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bdc6308d-a259-45db-90ea-094a8d1dd3a4/volumes/kubernetes.io~projected/kube-api-access-lztwd:{mountpoint:/var/lib/kubelet/pods/bdc6308d-a259-45db-90ea-094a8d1dd3a4/volumes/kubernetes.io~projected/kube-api-access-lztwd major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c31a736c-cd3e-42be-b783-fc25436b5302/volumes/kubernetes.io~projected/kube-api-access-vv2fd:{mountpoint:/var/lib/kubelet/pods/c31a736c-cd3e-42be-b783-fc25436b5302/volumes/kubernetes.io~projected/kube-api-access-vv2fd major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccf49970-fa36-4188-ab63-e14e440b73ec/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/ccf49970-fa36-4188-ab63-e14e440b73ec/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccf49970-fa36-4188-ab63-e14e440b73ec/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/ccf49970-fa36-4188-ab63-e14e440b73ec/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccf49970-fa36-4188-ab63-e14e440b73ec/volumes/kubernetes.io~projected/kube-api-access-45sk7:{mountpoint:/var/lib/kubelet/pods/ccf49970-fa36-4188-ab63-e14e440b73ec/volumes/kubernetes.io~projected/kube-api-access-45sk7 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d303600a-2390-4ba9-8ea8-a58bd505a109/volumes/kubernetes.io~projected/kube-api-access-wthz4:{mountpoint:/var/lib/kubelet/pods/d303600a-2390-4ba9-8ea8-a58bd505a109/volumes/kubernetes.io~projected/kube-api-access-wthz4 major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd15a503-2539-4458-90cb-dae3e0e28e62/volumes/kubernetes.io~projected/kube-api-access-kfdrz:{mountpoint:/var/lib/kubelet/pods/dd15a503-2539-4458-90cb-dae3e0e28e62/volumes/kubernetes.io~projected/kube-api-access-kfdrz major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0c1f2fe-ca1b-4556-8df1-134c67cee4f8/volumes/kubernetes.io~projected/kube-api-access-lk8vc:{mountpoint:/var/lib/kubelet/pods/e0c1f2fe-ca1b-4556-8df1-134c67cee4f8/volumes/kubernetes.io~projected/kube-api-access-lk8vc major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0c1f2fe-ca1b-4556-8df1-134c67cee4f8/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/e0c1f2fe-ca1b-4556-8df1-134c67cee4f8/volumes/kubernetes.io~secret/cert major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e297483a-0b86-4481-9597-0c89e9a6a7ac/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/e297483a-0b86-4481-9597-0c89e9a6a7ac/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e297483a-0b86-4481-9597-0c89e9a6a7ac/volumes/kubernetes.io~projected/kube-api-access-vcmwp:{mountpoint:/var/lib/kubelet/pods/e297483a-0b86-4481-9597-0c89e9a6a7ac/volumes/kubernetes.io~projected/kube-api-access-vcmwp major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e297483a-0b86-4481-9597-0c89e9a6a7ac/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/e297483a-0b86-4481-9597-0c89e9a6a7ac/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e297483a-0b86-4481-9597-0c89e9a6a7ac/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/e297483a-0b86-4481-9597-0c89e9a6a7ac/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea9c808c-62b4-4dc7-9761-bf6ee3700e34/volumes/kubernetes.io~projected/kube-api-access-j7gv7:{mountpoint:/var/lib/kubelet/pods/ea9c808c-62b4-4dc7-9761-bf6ee3700e34/volumes/kubernetes.io~projected/kube-api-access-j7gv7 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea9c808c-62b4-4dc7-9761-bf6ee3700e34/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ea9c808c-62b4-4dc7-9761-bf6ee3700e34/volumes/kubernetes.io~secret/serving-cert major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f14d2c99-6d70-44bf-9046-7cc1b84f1e7e/volumes/kubernetes.io~projected/kube-api-access-tmhwk:{mountpoint:/var/lib/kubelet/pods/f14d2c99-6d70-44bf-9046-7cc1b84f1e7e/volumes/kubernetes.io~projected/kube-api-access-tmhwk major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f14d2c99-6d70-44bf-9046-7cc1b84f1e7e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f14d2c99-6d70-44bf-9046-7cc1b84f1e7e/volumes/kubernetes.io~secret/serving-cert major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a/volumes/kubernetes.io~projected/kube-api-access-bptmv:{mountpoint:/var/lib/kubelet/pods/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a/volumes/kubernetes.io~projected/kube-api-access-bptmv major:0 minor:224 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/07a6bf522512eaff68e6934156e2b601822e62c986dd91f0a0c0bca2c7343d65/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/2e454d1d3e15957c8b213c6ff8de286e4814dc4b27706f96491ed82565c0a192/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/cde46844c2ffe8d3cbad175146c65d0a5322862dc9a485dc73d05e72140cd8b5/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/0a9937359fa40ba6959fdaadd1dd42c3f631270feda03e87660f8f91ae9551c6/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/496b911cf41f444bfc715127aa4b9b3b29cf5f4e91712e0511f08efe1853390c/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/cdcb6e8dcd2b3e2d619788bffc2f9c68e8cc15a4ae66dcc3ffac4782717af443/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/020f29f432b57efd636b55c562a0569dd032d4935eaa80bd0115e78446e68db6/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/d61865ddeb4a7c90a4a5b4f380026f3e9b72c196b26cd614783be62353fa18c1/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/fadfaf9245406558f8520c1d9afdd57dfe5e7490e401bd9d72d6c0d29e938222/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/8c7e116218642872f4c5ecd15efea1c613c86459875390fc0e1fc47d365993eb/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/d3f0a349b9ccbf249aca100cd20d54241c8423215513d83125e382d3da6eaef9/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/4bf4a080bfa2989a1671e8d297cc34274e42ba547b3eb7003446031b13238585/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/b6a36fc0561e9e3e58f3b3f47dc1c1238d470390b261197b53b251dbb2d3898a/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/111d576cc8d5d74257e9d77b9cabad84341c25dbfcde5a38513df563a6ef1e9e/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/1a05b20b5bd50d996b39d16563209099c35bdc615eeb133f8aa40585d7894be9/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/95f21036af8b4da97c4feddb68f89ca5bcf2532c591a5f9175c49be1acf9ca12/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/6878aa7d306acae6c000eb44a1633f5aea7f87d7f179eea5d0c94ae83f056ac7/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/e2216b4687f3b73f82d01a6880080b5257e06d1e81c538b31d0cecc8c0bd5d44/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/50376bd52f8a7f8a6921f835fce4dbcf87ca8c961a8666d97512ccd34b61689a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/50bd29a6e2e91a1a0b66e2c03ddf035d0d3008d24ed10c73097b7fa73b1185b8/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/699d492abb51bf3d091619c1f8f21bf6ffca2ae6caf1f2e37a0a6695233f4f15/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/2e1da93426cd828b68e1967eb8402a2e44cb17807828aa27849a3e58a9119824/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/6dc889c9bf108ee747bb27d5c12df056ee084514bae659c187accfd4b1f7c577/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/943790c51ffc7eef4fb86a732d99aa9ce9728e039ed6d2bc6f69aec5ca074e63/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/cf647e7ea59d56506b87b5e0aee84e1b2faf34fdaafd7276a990abfa5b782012/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/3ed6e91dafcdd6b34f78d81f9c4c3ff6d58eda4411834d72024d94e44d6fbf71/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/a008161d273642e158bc5a9337540f27a0e6847fe6f9e7e798567b464b364555/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/41216c5f6a0701f19b9d40894265370ea287e63a026b28fab71b6aa19d652002/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/6fc228c82d00ed9ed38b3c5ab33d8fc5339f99318f49833e6085a678a1af4fb9/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/67cc25fe0426bc8f42dcd6f8743147b11873317cfaac793108f08bdad097f95a/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/d5fdf8d753a3311e3ef02e687590dbe7086a8bc2c06be2984f3b70548ca7b533/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/41d81a56e586656765ae9cd7497520a7629fae234ff1c69cf8ca6dd2c9b5201e/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/491bd3470cb93cb68f3f54dc39a39ba513626c6064ceca212d5730d9f2ef9cc8/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/a0b8ee17794d856b77cf2846c064ac6615b7cd8b5c62743165b454145f050748/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/e33b70016ae8d488d2dda08c3508f25e5b5a93fb4b6887d34665074dd3d4df16/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/4c99612d32f6ed6b131d39f1add6edd2130e7099f337fb1c4afcf85cfef1d11f/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/3c5c1a347e8c6cebc897a5f4173a4bcd387df2f4cd44a1c51b5997306ad51ee3/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/eb4a5694d793aa654f984c1c523fe0d509436ac76ea5800a1c71688609472faf/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/05ea759ca1e94a0d97d57a86ac655a2ff5054a7ef8b6d8e91e731f32951b5355/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/25702c6c2a3d5a698550b547bfc9403d67df7e6beb49e22ad863f6a3cd8e7ede/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/9f0838e088e24e2d32d2101ca09099fa1340f5e51321fd5fb18cf17e8bd0442d/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/0497b4b8300d201a1dc280478f911c73e2a24753833d73ff29e15b494383115f/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/d458fb6dbd81779b9f49f3e020bee151d559cd1877f75c8f7706af91fa0276ba/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/ab5000c9fefb96d221fe04d7bdfecf0d90e649847049d4a028ca2692b8f4ba3e/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/1095cc4e603de33ba31df6adea3ad3f2e6ae42145f94b7fa27be3d41a07a2147/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/33d20b9ceebbe8af2b87e2cf88cb5f49fca453ee1cb8462634d98b978d763db5/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/00a900d5b91f861b2afac291fb85e07ad9267430f1acc4fc90ab74ba5ff7bb55/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/c4675b1d2655c77d4359c699fb6cd131073e156432ee11c7cf114340899f6052/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/0e03e035667ecc275608d69950f77e24c1cc1468d4fc1722af11cbd885e8a376/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/a00f9faf55d6b16bd7063711b23af0460741d64ca7c7e70cbc6a3c514ca6bfd0/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/42f8e72976c46b99cc71e43b0493ea4686118707b166cc6e21d69b927f742387/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/907ee6298d84160ef7d0b742e84f2a9406f0b43d3c60e3e0ec1380e3ea2d5b03/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/2094152e2644d351b5276f626af09a09566a19711f763f864f2a458ed0895183/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/621373c4340ad0b48c9b432d6b6679af9f7cd72d590e7a8514382d723e2bbee9/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 22:06:54.718740 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.718357    4889 manager.go:217] Machine: {Timestamp:2026-10-30 22:06:54.717785207 +0000 UTC m=+0.074151516 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099232 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29286363831a8798a39233637872d6 SystemUUID:ec292863-6383-1a87-98a3-9233637872d6 BootID:9fab111f-a068-424a-ac61-d6661241e36c Filesystems:[{Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6651a05-7ba7-484f-8490-f22665bcb84f/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:223 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/109c10c088184bcb0a693845404521cea84a57da050db42739d6f87e72638560/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/801d7671-e004-4da3-898e-664eb2f1b5c3/volumes/kubernetes.io~projected/kube-api-access-jrsqc DeviceMajor:0 DeviceMinor:61 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b84639d-6b5d-4795-a356-8755ea234585/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:218 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ea9c808c-62b4-4dc7-9761-bf6ee3700e34/volumes/kubernetes.io~projected/kube-api-access-j7gv7 DeviceMajor:0 DeviceMinor:225 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1e444015-5346-4106-93aa-ef43eb9d0be6/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e297483a-0b86-4481-9597-0c89e9a6a7ac/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:202 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f14d2c99-6d70-44bf-9046-7cc1b84f1e7e/volumes/kubernetes.io~projected/kube-api-access-tmhwk DeviceMajor:0 DeviceMinor:222 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f2b127804d27f49d0c9bed28f086e3bb999bbd9fa415e695653470e9034f375/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ec19290-25c4-4b52-8173-22c14e7c9056/volumes/kubernetes.io~projected/kube-api-access-5k4xc DeviceMajor:0 DeviceMinor:60 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ccf49970-fa36-4188-ab63-e14e440b73ec/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/dd15a503-2539-4458-90cb-dae3e0e28e62/volumes/kubernetes.io~projected/kube-api-access-kfdrz DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/5b84639d-6b5d-4795-a356-8755ea234585/volumes/kubernetes.io~projected/kube-api-access-92nhs DeviceMajor:0 DeviceMinor:227 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/784af23f4ecb79948e4642e53991b673c4b33f50f0bfb91aeb5ef7a88ca9ab7d/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9d1eecff-b22f-4864-a121-9fb507e783f2/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/20d5b724c1ef9e807e1f97ce33a81114cbb8ebca9d46488a059e263ff1a7e1ee/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f3a4ea6cd48a49887913384ad1cc06bef23dfe6263904754a20538780da8eea3/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0c1f2fe-ca1b-4556-8df1-134c67cee4f8/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:214 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da6d5bc0a53d2d71c94f4708be735cb287c3b7824e15690d1f3472fb87cb3453/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c31a736c-cd3e-42be-b783-fc25436b5302/volumes/kubernetes.io~projected/kube-api-access-vv2fd DeviceMajor:0 DeviceMinor:204 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/00b74f51-6c71-45f2-97af-6aea20b9eeaf/volumes/kubernetes.io~projected/kube-api-access-b66td DeviceMajor:0 DeviceMinor:209 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b8ee94be-1472-42e7-bf6e-d13e58671eeb/volumes/kubernetes.io~projected/kube-api-access-h7wm8 DeviceMajor:0 DeviceMinor:212 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3681ee3cc46929a2fb642aa02d8f7701443039483d0c39517a43ace21b066d3f/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4faf0685059f939c00321305f975604de2c8a02d6a234aa7fb7bd4792e0661d2/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3858b6612025b28ff98838f7ebf46418f61ea0e479908653b17a4d70c3bb3908/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/10d1cecc-9e30-4354-8b42-60ddceadff44/volumes/kubernetes.io~projected/kube-api-access-b4wvm DeviceMajor:0 DeviceMinor:62 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/41bec762b666a037acb1960a7ddf13317cfcc5a451f2d3c7aab97a47dd7bd6e4/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4049e53b-bffe-40fb-beef-22acc171315a/volumes/kubernetes.io~projected/kube-api-access-c7fw6 DeviceMajor:0 DeviceMinor:65 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/bdc6308d-a259-45db-90ea-094a8d1dd3a4/volumes/kubernetes.io~projected/kube-api-access-lztwd DeviceMajor:0 DeviceMinor:64 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e297483a-0b86-4481-9597-0c89e9a6a7ac/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:213 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e0c1f2fe-ca1b-4556-8df1-134c67cee4f8/volumes/kubernetes.io~projected/kube-api-access-lk8vc DeviceMajor:0 DeviceMinor:228 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea9c808c-62b4-4dc7-9761-bf6ee3700e34/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:215 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/471d2921a86a8eaa2f2bb973796671cf840e2862e459183ac3d4afe68995188b/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/605c1b60-1fa5-4257-a588-4b2e0a78bcca/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b6651a05-7ba7-484f-8490-f22665bcb84f/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:217 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ccf49970-fa36-4188-ab63-e14e440b73ec/volumes/kubernetes.io~projected/kube-api-access-45sk7 DeviceMajor:0 DeviceMinor:63 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/47dc8b20-b520-4f4d-bd48-74b78ba300ef/volumes/kubernetes.io~projected/kube-api-access-swswj DeviceMajor:0 DeviceMinor:66 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/88fe4dfd-dcda-444d-871b-24ca3fb10d77/volumes/kubernetes.io~projected/kube-api-access-xrm4r DeviceMajor:0 DeviceMinor:210 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3739156ea50346f67c1f1c70b6a7bd166cc518bbce04c34908a9c1148695a299/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/907de9fd04680f90043bccba46fefda4567401cf21b11ec468c4bc55d2492d6a/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b6651a05-7ba7-484f-8490-f22665bcb84f/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:219 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aa6e8f0edb6759c894052cf03580233f00ba2d529c93f92e36a4b1b9efe9f64b/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/605c1b60-1fa5-4257-a588-4b2e0a78bcca/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/200052bd-2f0a-46d2-8b9b-6bd250b17b05/volumes/kubernetes.io~projected/kube-api-access-h6cjm DeviceMajor:0 DeviceMinor:207 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b6651a05-7ba7-484f-8490-f22665bcb84f/volumes/kubernetes.io~projected/kube-api-access-vr8dq DeviceMajor:0 DeviceMinor:220 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/24c987a0-1cbf-4264-ba0e-586b738d08da/volumes/kubernetes.io~projected/kube-api-access-h824x DeviceMajor:0 DeviceMinor:221 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7d1343f6fe0cbba61605e7232037aa2743561125a07d7556dbdfd92daed9210c/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e297483a-0b86-4481-9597-0c89e9a6a7ac/volumes/kubernetes.io~projected/kube-api-access-vcmwp DeviceMajor:0 DeviceMinor:211 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/ccf49970-fa36-4188-ab63-e14e440b73ec/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/47dc8b20-b520-4f4d-bd48-74b78ba300ef/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e639ca449fd6481bd8538d41484639a47ad2d027e5d81ec2deb1b336f6c6c617/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e297483a-0b86-4481-9597-0c89e9a6a7ac/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9/volumes/kubernetes.io~projected/kube-api-access-k8ltk DeviceMajor:0 DeviceMinor:67 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/03750ea778f9d46aafdb422942074845d557e3711abaffa235c54ae20950287d/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/965e7b7f-9044-4c69-be4d-06ef9a1044e2/volumes/kubernetes.io~projected/kube-api-access-7qqhv DeviceMajor:0 DeviceMinor:208 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a/volumes/kubernetes.io~projected/kube-api-access-bptmv DeviceMajor:0 DeviceMinor:224 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/940a21805282e57baa73037cb6e8c0073b44aa3db60ee4e376faa4a835331ae7/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/965e7b7f-9044-4c69-be4d-06ef9a1044e2/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/473049b96ecb4925bb44684d88d7ed4a537e37099bdd407ebe98583432f67887/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eac8bfd37e67bf7de6667108361618f1e829410e6a381e7b137b3781e55f3ce4/userdata/shm DeviceMajor:0 DeviceMinor:295 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/052e2399-f78d-4eb2-9da7-e2b098f86da7/volumes/kubernetes.io~projected/kube-api-access-q5cps DeviceMajor:0 DeviceMinor:324 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3f403b34-f155-49e6-b327-8ad89d00bbc3/volumes/kubernetes.io~projected/kube-api-access-qdqmz DeviceMajor:0 DeviceMinor:205 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/605c1b60-1fa5-4257-a588-4b2e0a78bcca/volumes/kubernetes.io~projected/kube-api-access-dr2ww DeviceMajor:0 DeviceMinor:206 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/47dc8b20-b520-4f4d-bd48-74b78ba300ef/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/f14d2c99-6d70-44bf-9046-7cc1b84f1e7e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:216 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c543d14496ba1d58230b67dac2cad4b80d74f21e802d0bc59043dfd072cbcee4/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d303600a-2390-4ba9-8ea8-a58bd505a109/volumes/kubernetes.io~projected/kube-api-access-wthz4 DeviceMajor:0 DeviceMinor:293 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/293e7aaa41bae7e7f8f02cc4f11203729a2c7a3b847ff7e0b29819b5671d3a83/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:293e7aaa41bae7e MacAddress:52:a6:5c:c1:23:1c Speed:10000 Mtu:8901} {Name:3681ee3cc46929a MacAddress:ba:e3:fd:6a:8d:74 Speed:10000 Mtu:8901} {Name:471d2921a86a8ea MacAddress:72:6f:4b:85:07:1b Speed:10000 Mtu:8901} {Name:473049b96ecb492 MacAddress:2a:5d:f4:84:e2:28 Speed:10000 Mtu:8901} {Name:4f2b127804d27f4 MacAddress:82:ec:c6:2f:41:72 Speed:10000 Mtu:8901} {Name:4faf0685059f939 MacAddress:2a:c7:46:b0:1d:28 Speed:10000 Mtu:8901} {Name:784af23f4ecb799 MacAddress:46:ee:ff:a5:27:91 Speed:10000 Mtu:8901} {Name:7d1343f6fe0cbba MacAddress:16:4f:81:7e:3e:48 Speed:10000 Mtu:8901} {Name:aa6e8f0edb6759c MacAddress:ae:57:be:e2:fa:6e Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ea:b9:24:63 Speed:0 Mtu:9001} {Name:br-int MacAddress:0a:49:c3:ab:3a:47 Speed:0 Mtu:8901} {Name:c543d14496ba1d5 MacAddress:96:ec:8e:28:6c:e0 Speed:10000 Mtu:8901} {Name:da6d5bc0a53d2d7 MacAddress:32:38:e1:c5:68:bb Speed:10000 Mtu:8901} {Name:eac8bfd37e67bf7 MacAddress:96:c7:3a:5c:cf:2f Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ea:b9:24:63 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:aa:5c:7b:a4:3d:03 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:5a:e8:06:a5:ee:66 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 22:06:54.718740 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.718680    4889 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 22:06:54.718740 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.718725    4889 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 22:06:54.719028 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.719003    4889 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 22:06:54.719153 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.719028    4889 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-185.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 22:06:54.719192 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.719156    4889 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 22:06:54.719192 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.719164    4889 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 22:06:54.719192 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.719177    4889 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 22:06:54.719192 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.719191    4889 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 22:06:54.719365 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.719210    4889 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 22:06:54.719404 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.719394    4889 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 22:06:54.719502 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.719493    4889 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 22:06:54.719537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.719529    4889 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 22:06:54.719564 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.719540    4889 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 22:06:54.719564 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.719550    4889 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 22:06:54.719564 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.719559    4889 kubelet.go:405] "Adding apiserver pod source"
Oct 30 22:06:54.719654 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.719568    4889 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 22:06:54.720308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.720296    4889 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 22:06:54.720308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.720308    4889 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 22:06:54.720965 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.720920    4889 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 22:06:54.724390 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.724373    4889 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 22:06:54.724390 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.724388    4889 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 22:06:54.724518 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.724494    4889 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 22:06:54.724714 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.724702    4889 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 22:06:54.724745 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.724732    4889 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 22:06:54.724826 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.724817    4889 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 22:06:54.724862 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.724832    4889 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 22:06:54.724862 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.724839    4889 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 22:06:54.724862 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.724847    4889 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 22:06:54.724862 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.724854    4889 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 22:06:54.724862 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.724861    4889 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 22:06:54.725060 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.724866    4889 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 22:06:54.725060 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.724872    4889 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 22:06:54.725060 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.724878    4889 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 22:06:54.725060 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.724883    4889 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 22:06:54.725060 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.724891    4889 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 22:06:54.725060 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.724901    4889 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 22:06:54.725060 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.724925    4889 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 22:06:54.725060 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.724930    4889 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 22:06:54.728481 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.728353    4889 server.go:1263] "Started kubelet"
Oct 30 22:06:54.728481 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.728417    4889 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 22:06:54.728776 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.728677    4889 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 22:06:54.728776 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.728741    4889 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 22:06:54.728925 ip-10-0-142-185 systemd[1]: Started Kubernetes Kubelet.
Oct 30 22:06:54.729094 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.729055    4889 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 22:06:54.730308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.730268    4889 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 22:06:54.731965 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.731879    4889 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 22:06:54.731965 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.731914    4889 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 22:06:54.732514 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:54.731951    4889 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-185.ec2.internal\" not found"
Oct 30 22:06:54.732514 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.732024    4889 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 22:06:54.732514 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.732004    4889 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 22:01:15 +0000 UTC" deadline="2028-05-14 09:02:27.398684113 +0000 UTC"
Oct 30 22:06:54.732514 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.732031    4889 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 22:06:54.732514 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.732101    4889 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 22:06:54.732514 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.732036    4889 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13474h55m32.66665094s"
Oct 30 22:06:54.734250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734201    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="605c1b60-1fa5-4257-a588-4b2e0a78bcca" volumeName="kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-default-certificate" seLinuxMountContext=""
Oct 30 22:06:54.734316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734252    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdc6308d-a259-45db-90ea-094a8d1dd3a4" volumeName="kubernetes.io/configmap/bdc6308d-a259-45db-90ea-094a8d1dd3a4-cni-binary-copy" seLinuxMountContext=""
Oct 30 22:06:54.734316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734261    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccf49970-fa36-4188-ab63-e14e440b73ec" volumeName="kubernetes.io/empty-dir/ccf49970-fa36-4188-ab63-e14e440b73ec-tmp" seLinuxMountContext=""
Oct 30 22:06:54.734316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734270    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea9c808c-62b4-4dc7-9761-bf6ee3700e34" volumeName="kubernetes.io/projected/ea9c808c-62b4-4dc7-9761-bf6ee3700e34-kube-api-access-j7gv7" seLinuxMountContext=""
Oct 30 22:06:54.734316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734277    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f14d2c99-6d70-44bf-9046-7cc1b84f1e7e" volumeName="kubernetes.io/projected/f14d2c99-6d70-44bf-9046-7cc1b84f1e7e-kube-api-access-tmhwk" seLinuxMountContext=""
Oct 30 22:06:54.734316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734284    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ec19290-25c4-4b52-8173-22c14e7c9056" volumeName="kubernetes.io/empty-dir/2ec19290-25c4-4b52-8173-22c14e7c9056-tmp-dir" seLinuxMountContext=""
Oct 30 22:06:54.734316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734292    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c31a736c-cd3e-42be-b783-fc25436b5302" volumeName="kubernetes.io/configmap/c31a736c-cd3e-42be-b783-fc25436b5302-config-volume" seLinuxMountContext=""
Oct 30 22:06:54.734316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734298    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e297483a-0b86-4481-9597-0c89e9a6a7ac" volumeName="kubernetes.io/configmap/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-certificates" seLinuxMountContext=""
Oct 30 22:06:54.734316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734306    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00b74f51-6c71-45f2-97af-6aea20b9eeaf" volumeName="kubernetes.io/projected/00b74f51-6c71-45f2-97af-6aea20b9eeaf-kube-api-access-b66td" seLinuxMountContext=""
Oct 30 22:06:54.734316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734313    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4049e53b-bffe-40fb-beef-22acc171315a" volumeName="kubernetes.io/projected/4049e53b-bffe-40fb-beef-22acc171315a-kube-api-access-c7fw6" seLinuxMountContext=""
Oct 30 22:06:54.734316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734320    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="605c1b60-1fa5-4257-a588-4b2e0a78bcca" volumeName="kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-stats-auth" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734327    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88fe4dfd-dcda-444d-871b-24ca3fb10d77" volumeName="kubernetes.io/projected/88fe4dfd-dcda-444d-871b-24ca3fb10d77-kube-api-access-xrm4r" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734334    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6651a05-7ba7-484f-8490-f22665bcb84f" volumeName="kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-bound-sa-token" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734348    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f14d2c99-6d70-44bf-9046-7cc1b84f1e7e" volumeName="kubernetes.io/configmap/f14d2c99-6d70-44bf-9046-7cc1b84f1e7e-config" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734357    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="200052bd-2f0a-46d2-8b9b-6bd250b17b05" volumeName="kubernetes.io/configmap/200052bd-2f0a-46d2-8b9b-6bd250b17b05-telemetry-config" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734365    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="200052bd-2f0a-46d2-8b9b-6bd250b17b05" volumeName="kubernetes.io/projected/200052bd-2f0a-46d2-8b9b-6bd250b17b05-kube-api-access-h6cjm" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734371    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b84639d-6b5d-4795-a356-8755ea234585" volumeName="kubernetes.io/configmap/5b84639d-6b5d-4795-a356-8755ea234585-config" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734378    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="965e7b7f-9044-4c69-be4d-06ef9a1044e2" volumeName="kubernetes.io/configmap/965e7b7f-9044-4c69-be4d-06ef9a1044e2-service-ca-bundle" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734386    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="965e7b7f-9044-4c69-be4d-06ef9a1044e2" volumeName="kubernetes.io/secret/965e7b7f-9044-4c69-be4d-06ef9a1044e2-serving-cert" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734394    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdc6308d-a259-45db-90ea-094a8d1dd3a4" volumeName="kubernetes.io/configmap/bdc6308d-a259-45db-90ea-094a8d1dd3a4-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734402    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0c1f2fe-ca1b-4556-8df1-134c67cee4f8" volumeName="kubernetes.io/projected/e0c1f2fe-ca1b-4556-8df1-134c67cee4f8-kube-api-access-lk8vc" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734414    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10d1cecc-9e30-4354-8b42-60ddceadff44" volumeName="kubernetes.io/configmap/10d1cecc-9e30-4354-8b42-60ddceadff44-serviceca" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734422    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="801d7671-e004-4da3-898e-664eb2f1b5c3" volumeName="kubernetes.io/projected/801d7671-e004-4da3-898e-664eb2f1b5c3-kube-api-access-jrsqc" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734429    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e297483a-0b86-4481-9597-0c89e9a6a7ac" volumeName="kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-bound-sa-token" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734439    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9" volumeName="kubernetes.io/projected/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-kube-api-access-k8ltk" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734447    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="605c1b60-1fa5-4257-a588-4b2e0a78bcca" volumeName="kubernetes.io/projected/605c1b60-1fa5-4257-a588-4b2e0a78bcca-kube-api-access-dr2ww" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734454    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="801d7671-e004-4da3-898e-664eb2f1b5c3" volumeName="kubernetes.io/configmap/801d7671-e004-4da3-898e-664eb2f1b5c3-iptables-alerter-script" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734462    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6651a05-7ba7-484f-8490-f22665bcb84f" volumeName="kubernetes.io/configmap/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-certificates" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734472    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccf49970-fa36-4188-ab63-e14e440b73ec" volumeName="kubernetes.io/projected/ccf49970-fa36-4188-ab63-e14e440b73ec-kube-api-access-45sk7" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734487    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a" volumeName="kubernetes.io/configmap/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-config" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734495    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9" volumeName="kubernetes.io/configmap/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-multus-daemon-config" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734502    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="965e7b7f-9044-4c69-be4d-06ef9a1044e2" volumeName="kubernetes.io/projected/965e7b7f-9044-4c69-be4d-06ef9a1044e2-kube-api-access-7qqhv" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734508    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea9c808c-62b4-4dc7-9761-bf6ee3700e34" volumeName="kubernetes.io/secret/ea9c808c-62b4-4dc7-9761-bf6ee3700e34-serving-cert" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734515    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="052e2399-f78d-4eb2-9da7-e2b098f86da7" volumeName="kubernetes.io/projected/052e2399-f78d-4eb2-9da7-e2b098f86da7-kube-api-access-q5cps" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734523    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47dc8b20-b520-4f4d-bd48-74b78ba300ef" volumeName="kubernetes.io/configmap/47dc8b20-b520-4f4d-bd48-74b78ba300ef-env-overrides" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734533    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47dc8b20-b520-4f4d-bd48-74b78ba300ef" volumeName="kubernetes.io/secret/47dc8b20-b520-4f4d-bd48-74b78ba300ef-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734540    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b84639d-6b5d-4795-a356-8755ea234585" volumeName="kubernetes.io/projected/5b84639d-6b5d-4795-a356-8755ea234585-kube-api-access-92nhs" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734547    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6651a05-7ba7-484f-8490-f22665bcb84f" volumeName="kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-kube-api-access-vr8dq" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734554    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6651a05-7ba7-484f-8490-f22665bcb84f" volumeName="kubernetes.io/secret/b6651a05-7ba7-484f-8490-f22665bcb84f-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734561    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8ee94be-1472-42e7-bf6e-d13e58671eeb" volumeName="kubernetes.io/projected/b8ee94be-1472-42e7-bf6e-d13e58671eeb-kube-api-access-h7wm8" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734567    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c31a736c-cd3e-42be-b783-fc25436b5302" volumeName="kubernetes.io/projected/c31a736c-cd3e-42be-b783-fc25436b5302-kube-api-access-vv2fd" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734574    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="965e7b7f-9044-4c69-be4d-06ef9a1044e2" volumeName="kubernetes.io/empty-dir/965e7b7f-9044-4c69-be4d-06ef9a1044e2-tmp" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734582    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c31a736c-cd3e-42be-b783-fc25436b5302" volumeName="kubernetes.io/empty-dir/c31a736c-cd3e-42be-b783-fc25436b5302-tmp-dir" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734589    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d303600a-2390-4ba9-8ea8-a58bd505a109" volumeName="kubernetes.io/projected/d303600a-2390-4ba9-8ea8-a58bd505a109-kube-api-access-wthz4" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734596    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e297483a-0b86-4481-9597-0c89e9a6a7ac" volumeName="kubernetes.io/configmap/e297483a-0b86-4481-9597-0c89e9a6a7ac-trusted-ca" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734603    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e297483a-0b86-4481-9597-0c89e9a6a7ac" volumeName="kubernetes.io/secret/e297483a-0b86-4481-9597-0c89e9a6a7ac-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734610    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6651a05-7ba7-484f-8490-f22665bcb84f" volumeName="kubernetes.io/empty-dir/b6651a05-7ba7-484f-8490-f22665bcb84f-ca-trust-extracted" seLinuxMountContext=""
Oct 30 22:06:54.734608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734617    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e297483a-0b86-4481-9597-0c89e9a6a7ac" volumeName="kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-kube-api-access-vcmwp" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734813    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea4b0705-2096-41da-b78b-b62590f236e8" volumeName="kubernetes.io/configmap/ea4b0705-2096-41da-b78b-b62590f236e8-nginx-conf" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734834    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="052e2399-f78d-4eb2-9da7-e2b098f86da7" volumeName="kubernetes.io/empty-dir/052e2399-f78d-4eb2-9da7-e2b098f86da7-data-volume" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734860    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f403b34-f155-49e6-b327-8ad89d00bbc3" volumeName="kubernetes.io/projected/3f403b34-f155-49e6-b327-8ad89d00bbc3-kube-api-access-qdqmz" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734881    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="965e7b7f-9044-4c69-be4d-06ef9a1044e2" volumeName="kubernetes.io/empty-dir/965e7b7f-9044-4c69-be4d-06ef9a1044e2-snapshots" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734894    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e297483a-0b86-4481-9597-0c89e9a6a7ac" volumeName="kubernetes.io/empty-dir/e297483a-0b86-4481-9597-0c89e9a6a7ac-ca-trust-extracted" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.734913    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d1eecff-b22f-4864-a121-9fb507e783f2" volumeName="kubernetes.io/secret/9d1eecff-b22f-4864-a121-9fb507e783f2-agent-certs" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.735011    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdc6308d-a259-45db-90ea-094a8d1dd3a4" volumeName="kubernetes.io/projected/bdc6308d-a259-45db-90ea-094a8d1dd3a4-kube-api-access-lztwd" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.735025    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0c1f2fe-ca1b-4556-8df1-134c67cee4f8" volumeName="kubernetes.io/secret/e0c1f2fe-ca1b-4556-8df1-134c67cee4f8-cert" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.735064    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a" volumeName="kubernetes.io/projected/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-kube-api-access-bptmv" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.735077    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ec19290-25c4-4b52-8173-22c14e7c9056" volumeName="kubernetes.io/projected/2ec19290-25c4-4b52-8173-22c14e7c9056-kube-api-access-5k4xc" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.735094    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6651a05-7ba7-484f-8490-f22665bcb84f" volumeName="kubernetes.io/secret/b6651a05-7ba7-484f-8490-f22665bcb84f-installation-pull-secrets" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.735108    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccf49970-fa36-4188-ab63-e14e440b73ec" volumeName="kubernetes.io/empty-dir/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-tuned" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.735121    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e444015-5346-4106-93aa-ef43eb9d0be6" volumeName="kubernetes.io/secret/1e444015-5346-4106-93aa-ef43eb9d0be6-original-pull-secret" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.735138    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24c987a0-1cbf-4264-ba0e-586b738d08da" volumeName="kubernetes.io/projected/24c987a0-1cbf-4264-ba0e-586b738d08da-kube-api-access-h824x" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.735150    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47dc8b20-b520-4f4d-bd48-74b78ba300ef" volumeName="kubernetes.io/configmap/47dc8b20-b520-4f4d-bd48-74b78ba300ef-ovnkube-config" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.735166    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b84639d-6b5d-4795-a356-8755ea234585" volumeName="kubernetes.io/secret/5b84639d-6b5d-4795-a356-8755ea234585-serving-cert" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.735178    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d1eecff-b22f-4864-a121-9fb507e783f2" volumeName="kubernetes.io/configmap/9d1eecff-b22f-4864-a121-9fb507e783f2-konnectivity-ca" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.735190    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e297483a-0b86-4481-9597-0c89e9a6a7ac" volumeName="kubernetes.io/secret/e297483a-0b86-4481-9597-0c89e9a6a7ac-installation-pull-secrets" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.735207    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47dc8b20-b520-4f4d-bd48-74b78ba300ef" volumeName="kubernetes.io/projected/47dc8b20-b520-4f4d-bd48-74b78ba300ef-kube-api-access-swswj" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.735242    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="965e7b7f-9044-4c69-be4d-06ef9a1044e2" volumeName="kubernetes.io/configmap/965e7b7f-9044-4c69-be4d-06ef9a1044e2-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.735260    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea9c808c-62b4-4dc7-9761-bf6ee3700e34" volumeName="kubernetes.io/configmap/ea9c808c-62b4-4dc7-9761-bf6ee3700e34-config" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.735273    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea9c808c-62b4-4dc7-9761-bf6ee3700e34" volumeName="kubernetes.io/configmap/ea9c808c-62b4-4dc7-9761-bf6ee3700e34-trusted-ca" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.735285    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f14d2c99-6d70-44bf-9046-7cc1b84f1e7e" volumeName="kubernetes.io/secret/f14d2c99-6d70-44bf-9046-7cc1b84f1e7e-serving-cert" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.735302    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10d1cecc-9e30-4354-8b42-60ddceadff44" volumeName="kubernetes.io/projected/10d1cecc-9e30-4354-8b42-60ddceadff44-kube-api-access-b4wvm" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.735314    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6651a05-7ba7-484f-8490-f22665bcb84f" volumeName="kubernetes.io/configmap/b6651a05-7ba7-484f-8490-f22665bcb84f-trusted-ca" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.735331    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdc6308d-a259-45db-90ea-094a8d1dd3a4" volumeName="kubernetes.io/configmap/bdc6308d-a259-45db-90ea-094a8d1dd3a4-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.735344    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd15a503-2539-4458-90cb-dae3e0e28e62" volumeName="kubernetes.io/projected/dd15a503-2539-4458-90cb-dae3e0e28e62-kube-api-access-kfdrz" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.735356    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47dc8b20-b520-4f4d-bd48-74b78ba300ef" volumeName="kubernetes.io/configmap/47dc8b20-b520-4f4d-bd48-74b78ba300ef-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.735371    4889 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9" volumeName="kubernetes.io/configmap/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-cni-binary-copy" seLinuxMountContext=""
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.735384    4889 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.735391    4889 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.736153    4889 factory.go:55] Registering systemd factory
Oct 30 22:06:54.736604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.736170    4889 factory.go:223] Registration of the systemd container factory successfully
Oct 30 22:06:54.738586 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.738490    4889 factory.go:153] Registering CRI-O factory
Oct 30 22:06:54.738586 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.738516    4889 factory.go:223] Registration of the crio container factory successfully
Oct 30 22:06:54.738586 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.738585    4889 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 22:06:54.741528 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.738606    4889 factory.go:103] Registering Raw factory
Oct 30 22:06:54.741528 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.738623    4889 manager.go:1199] Started watching for new ooms in manager
Oct 30 22:06:54.741528 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.739480    4889 manager.go:319] Starting recovery of all containers
Oct 30 22:06:54.750306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.750288    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/1.log"
Oct 30 22:06:54.750402 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.750326    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/1.log"
Oct 30 22:06:54.764265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.764193    4889 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 22:06:54.766081 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.766030    4889 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 22:06:54.766081 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.766083    4889 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 22:06:54.766287 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.766112    4889 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 22:06:54.766287 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:54.766173    4889 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 22:06:54.780918 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.780889    4889 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 22:06:54.781074 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.781030    4889 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:06:54.782198 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.782176    4889 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:06:54.784401 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.784378    4889 generic.go:356] "Generic (PLEG): container finished" podID="bdc6308d-a259-45db-90ea-094a8d1dd3a4" containerID="0fc65ce8274391c56c68dbc5d10118c05596ff8078e4775ab54ad285ed0aa7af" exitCode=0
Oct 30 22:06:54.784401 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.784399    4889 generic.go:356] "Generic (PLEG): container finished" podID="bdc6308d-a259-45db-90ea-094a8d1dd3a4" containerID="a680883a4e8889b474b8c6bde3333104e153f9c182b78fa821b4477df7cd6894" exitCode=0
Oct 30 22:06:54.784595 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.784405    4889 generic.go:356] "Generic (PLEG): container finished" podID="bdc6308d-a259-45db-90ea-094a8d1dd3a4" containerID="fb9521f9329cfdc5b9b4f674b8bbc068e8693e324184b9c590b97f787317ecdc" exitCode=0
Oct 30 22:06:54.784595 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.784411    4889 generic.go:356] "Generic (PLEG): container finished" podID="bdc6308d-a259-45db-90ea-094a8d1dd3a4" containerID="bf22235e056f0bc95aff5bafa0dc1df8420e1cf1182ae2a5089e6fc9ce8cf198" exitCode=0
Oct 30 22:06:54.784595 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.784418    4889 generic.go:356] "Generic (PLEG): container finished" podID="bdc6308d-a259-45db-90ea-094a8d1dd3a4" containerID="13789b38308d9965add9531d538c51cafadcc52906304558f308b7084b3c5ffc" exitCode=0
Oct 30 22:06:54.784595 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.784426    4889 generic.go:356] "Generic (PLEG): container finished" podID="bdc6308d-a259-45db-90ea-094a8d1dd3a4" containerID="172e6bd2a08937872132948b1506e962b0bbfb589b773266647184426126fea8" exitCode=0
Oct 30 22:06:54.793604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.793570    4889 generic.go:356] "Generic (PLEG): container finished" podID="c3e2926fda086fe0743710e38cf91711" containerID="dc37d4cca2b85424065882973fbb214ed65b63b822dc095e27375a130e7fe443" exitCode=0
Oct 30 22:06:54.794719 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.794690    4889 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:06:54.795188 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.795165    4889 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:06:54.820316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.820290    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/1.log"
Oct 30 22:06:54.820419 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.820325    4889 generic.go:356] "Generic (PLEG): container finished" podID="ea9c808c-62b4-4dc7-9761-bf6ee3700e34" containerID="d60d0383fb24f739b8698d09dfc099d55168ef8507ac722ceb2547384d90c615" exitCode=255
Oct 30 22:06:54.832245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.832209    4889 manager.go:324] Recovery completed
Oct 30 22:06:54.850653 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.850591    4889 cpu_manager.go:226] "Starting" policy="none"
Oct 30 22:06:54.850653 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.850606    4889 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 22:06:54.850653 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.850621    4889 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 22:06:54.850818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.850738    4889 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 22:06:54.850818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.850749    4889 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 22:06:54.850818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.850767    4889 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 22:06:54.850818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.850772    4889 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 22:06:54.850818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.850777    4889 policy_none.go:50] "Start"
Oct 30 22:06:54.850818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.850785    4889 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 22:06:54.850818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.850792    4889 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 22:06:54.851001 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.850914    4889 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 22:06:54.851001 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.850921    4889 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 22:06:54.851001 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.850927    4889 policy_none.go:44] "Start"
Oct 30 22:06:54.855124 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.855096    4889 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 22:06:54.857810 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.857791    4889 manager.go:341] "Starting Device Plugin manager"
Oct 30 22:06:54.858314 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:54.857822    4889 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 22:06:54.858369 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.858318    4889 server.go:87] "Starting device plugin registration server"
Oct 30 22:06:54.858564 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.858555    4889 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 22:06:54.858600 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.858566    4889 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 22:06:54.858695 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.858679    4889 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 22:06:54.858802 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.858769    4889 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 22:06:54.858873 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.858828    4889 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 22:06:54.858873 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.858835    4889 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 22:06:54.859006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.858904    4889 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T22:06:54.858799406Z","UUID":"cbe9610a-cd38-457c-8d5a-b77b903a3135","Handler":null,"Name":"","Endpoint":""}
Oct 30 22:06:54.861628 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:54.860940    4889 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 22:06:54.861628 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.861023    4889 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 22:06:54.861628 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.861066    4889 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 22:06:54.866305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.866274    4889 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-185.ec2.internal"]
Oct 30 22:06:54.866901 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.866885    4889 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:54.866991 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.866956    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal" event={"ID":"c3e2926fda086fe0743710e38cf91711","Type":"ContainerStarted","Data":"77c95add65b87af616cd566bbbf5da6ddc3d60cb5bdc96ee8d2382bb9dc1cf8b"}
Oct 30 22:06:54.867054 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.867002    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal" event={"ID":"c3e2926fda086fe0743710e38cf91711","Type":"ContainerDied","Data":"dc37d4cca2b85424065882973fbb214ed65b63b822dc095e27375a130e7fe443"}
Oct 30 22:06:54.867054 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.867025    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal" event={"ID":"c3e2926fda086fe0743710e38cf91711","Type":"ContainerStarted","Data":"940a21805282e57baa73037cb6e8c0073b44aa3db60ee4e376faa4a835331ae7"}
Oct 30 22:06:54.867156 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.867082    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-185.ec2.internal" event={"ID":"496cd66cfb3bc1cef71f828d2683a76e","Type":"ContainerStarted","Data":"050605e2bb6fc90d8898833cc2f8ea6c6261849a2bdd0d0e57e1264527992606"}
Oct 30 22:06:54.867156 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.867097    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-185.ec2.internal" event={"ID":"496cd66cfb3bc1cef71f828d2683a76e","Type":"ContainerStarted","Data":"109c10c088184bcb0a693845404521cea84a57da050db42739d6f87e72638560"}
Oct 30 22:06:54.867274 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.867154    4889 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:54.890152 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.890131    4889 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:06:54.890271 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:54.890179    4889 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-185.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:54.890412 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.890400    4889 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:06:54.890446 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:54.890431    4889 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:54.959475 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.959449    4889 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:06:54.960682 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.960665    4889 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 22:06:54.960773 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.960695    4889 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 22:06:54.960773 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.960705    4889 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-185.ec2.internal" event="NodeHasSufficientPID"
Oct 30 22:06:54.960773 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.960745    4889 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-185.ec2.internal"
Oct 30 22:06:54.981983 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.981963    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3e2926fda086fe0743710e38cf91711-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal\" (UID: \"c3e2926fda086fe0743710e38cf91711\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:54.982056 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.981988    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/496cd66cfb3bc1cef71f828d2683a76e-config\") pod \"kube-apiserver-proxy-ip-10-0-142-185.ec2.internal\" (UID: \"496cd66cfb3bc1cef71f828d2683a76e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:54.982109 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.982092    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3e2926fda086fe0743710e38cf91711-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal\" (UID: \"c3e2926fda086fe0743710e38cf91711\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:54.998632 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.998612    4889 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-142-185.ec2.internal"
Oct 30 22:06:54.998720 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:54.998670    4889 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-185.ec2.internal"
Oct 30 22:06:55.082929 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.082895    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/496cd66cfb3bc1cef71f828d2683a76e-config\") pod \"kube-apiserver-proxy-ip-10-0-142-185.ec2.internal\" (UID: \"496cd66cfb3bc1cef71f828d2683a76e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:55.082929 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.082928    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3e2926fda086fe0743710e38cf91711-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal\" (UID: \"c3e2926fda086fe0743710e38cf91711\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:55.083122 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.082948    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3e2926fda086fe0743710e38cf91711-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal\" (UID: \"c3e2926fda086fe0743710e38cf91711\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:55.083122 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.082996    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3e2926fda086fe0743710e38cf91711-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal\" (UID: \"c3e2926fda086fe0743710e38cf91711\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:55.083122 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.083006    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/496cd66cfb3bc1cef71f828d2683a76e-config\") pod \"kube-apiserver-proxy-ip-10-0-142-185.ec2.internal\" (UID: \"496cd66cfb3bc1cef71f828d2683a76e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:55.083122 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.083006    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3e2926fda086fe0743710e38cf91711-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal\" (UID: \"c3e2926fda086fe0743710e38cf91711\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:55.720767 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.720735    4889 apiserver.go:52] "Watching apiserver"
Oct 30 22:06:55.775618 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.775583    4889 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 22:06:55.777652 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.777614    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw","kube-system/kas-connection-checker-685895c7c9-dvcrc","kube-system/konnectivity-agent-2fhd5","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jh5sc","openshift-network-diagnostics/network-check-target-zcl2w","openshift-insights/insights-operator-7c4986d449-vwshh","openshift-ingress-canary/ingress-canary-p89nn","openshift-dns/dns-default-t2gq6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-8kfh5","openshift-multus/multus-additional-cni-plugins-v69rv","openshift-ingress/router-default-fb457cdb7-m99g8","openshift-cluster-node-tuning-operator/tuned-p4g27","openshift-multus/network-metrics-daemon-fvsmv","openshift-insights/insights-runtime-extractor-mwjsl","openshift-network-console/networking-console-plugin-86c478bb88-n42rm","kube-system/kube-apiserver-proxy-ip-10-0-142-185.ec2.internal","openshift-dns/node-resolver-9dqlz","openshift-image-registry/image-registry-54b978dcc8-sch2b","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-48hjx","openshift-multus/multus-p8lsb","kube-system/kas-connection-checker-685895c7c9-74sl2","kube-system/kas-connection-checker-685895c7c9-k8lw5","openshift-image-registry/node-ca-nwtzc","kube-system/global-pull-secret-syncer-ljnhp","openshift-console-operator/console-operator-7cd56b8dbc-qbz82","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v","openshift-image-registry/image-registry-6b9d7c7dff-82x2r","openshift-service-ca-operator/service-ca-operator-544d749c8c-jd958","openshift-network-diagnostics/network-check-source-6f78d59f99-9xdrk","openshift-network-operator/iptables-alerter-86vll"]
Oct 30 22:06:55.779291 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.779269    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:55.780207 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.780184    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"
Oct 30 22:06:55.780496 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.780475    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"
Oct 30 22:06:55.781501 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.781479    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:55.782296 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.782281    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-n42rm"
Oct 30 22:06:55.783082 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.783065    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:55.788653 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.788636    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:55.789201 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.789188    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:55.790397 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.790380    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:55.798457 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.798434    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 22:06:55.798457 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.798450    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 22:06:55.798457 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.798437    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 22:06:55.798457 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.798438    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 22:06:55.798689 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.798613    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:06:55.798689 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.798642    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 22:06:55.798840 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.798823    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 22:06:55.798894 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.798825    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 22:06:55.798894 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.798862    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 22:06:55.798894 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.798872    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 22:06:55.798894 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.798879    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 22:06:55.798894 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.798862    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 22:06:55.798894 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.798891    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 22:06:55.799111 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.798890    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:06:55.799111 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.798906    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 22:06:55.799111 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.798928    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 22:06:55.799111 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.798951    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 22:06:55.804347 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.804310    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 22:06:55.804459 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.804443    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 22:06:55.804524 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.804510    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 22:06:55.804578 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.804565    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 22:06:55.804687 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.804665    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 22:06:55.804779 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.804719    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:06:55.804779 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.804728    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 22:06:55.805003 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.804987    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 22:06:55.805078 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.805059    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:06:55.805133 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.805088    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 22:06:55.805133 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.805094    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 22:06:55.805212 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.805136    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 22:06:55.805297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.805271    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 22:06:55.805492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.805309    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 22:06:55.805492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.805411    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 22:06:55.805492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.805428    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 22:06:55.805492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.805470    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 22:06:55.805815 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.805795    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 22:06:55.805890 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.805816    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 22:06:55.806023 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.806007    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 22:06:55.806023 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.806013    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 22:06:55.806423 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.806404    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 22:06:55.806488 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.806433    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 22:06:55.806488 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.806473    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 22:06:55.806488 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.806477    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 22:06:55.806635 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.806499    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 22:06:55.806635 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.806531    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:06:55.806635 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.806538    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 22:06:55.810887 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.810870    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 22:06:55.810960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.810904    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 22:06:55.810960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.810917    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:06:55.811025 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.811008    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 22:06:55.811025 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.811016    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 22:06:55.811134 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.811050    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 22:06:55.811134 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.811064    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 22:06:55.811134 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.811070    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 22:06:55.811134 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.811085    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 22:06:55.811134 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.811108    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 22:06:55.811134 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.811116    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:06:55.811134 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.811133    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 22:06:55.811374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.811140    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 22:06:55.811374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.811175    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 22:06:55.817831 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.817817    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 22:06:55.818417 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.818404    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 22:06:55.819262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.819249    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 22:06:55.819440 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.819425    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 22:06:55.823725 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.823709    4889 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:55.823888 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.823876    4889 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:55.834715 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.834701    4889 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 22:06:55.846731 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.846715    4889 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:06:55.846780 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:55.846752    4889 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-185.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:55.847342 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.847329    4889 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:06:55.847380 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:55.847367    4889 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-185.ec2.internal"
Oct 30 22:06:55.886866 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.886839    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccf49970-fa36-4188-ab63-e14e440b73ec-tmp\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.886866 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.886866    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b6651a05-7ba7-484f-8490-f22665bcb84f-ca-trust-extracted\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:55.887037 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.886887    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b66td\" (UniqueName: \"kubernetes.io/projected/00b74f51-6c71-45f2-97af-6aea20b9eeaf-kube-api-access-b66td\") pod \"kas-connection-checker-685895c7c9-74sl2\" (UID: \"00b74f51-6c71-45f2-97af-6aea20b9eeaf\") " pod="kube-system/kas-connection-checker-685895c7c9-74sl2"
Oct 30 22:06:55.887037 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.886909    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-systemd\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.887037 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.886970    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6651a05-7ba7-484f-8490-f22665bcb84f-trusted-ca\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:55.887037 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.886995    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b6651a05-7ba7-484f-8490-f22665bcb84f-ca-trust-extracted\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:55.887037 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.887033    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h824x\" (UniqueName: \"kubernetes.io/projected/24c987a0-1cbf-4264-ba0e-586b738d08da-kube-api-access-h824x\") pod \"network-check-source-6f78d59f99-9xdrk\" (UID: \"24c987a0-1cbf-4264-ba0e-586b738d08da\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9xdrk"
Oct 30 22:06:55.887281 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.887055    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-cni-bin\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.887281 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.887113    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/965e7b7f-9044-4c69-be4d-06ef9a1044e2-serving-cert\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:55.887281 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.887121    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccf49970-fa36-4188-ab63-e14e440b73ec-tmp\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.887864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.887175    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b6651a05-7ba7-484f-8490-f22665bcb84f-image-registry-private-configuration\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:55.887864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.887714    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-kubernetes\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.887864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.887794    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6651a05-7ba7-484f-8490-f22665bcb84f-trusted-ca\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:55.888071 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.887937    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7wm8\" (UniqueName: \"kubernetes.io/projected/b8ee94be-1472-42e7-bf6e-d13e58671eeb-kube-api-access-h7wm8\") pod \"kas-connection-checker-685895c7c9-dvcrc\" (UID: \"b8ee94be-1472-42e7-bf6e-d13e58671eeb\") " pod="kube-system/kas-connection-checker-685895c7c9-dvcrc"
Oct 30 22:06:55.888071 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.887975    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7gv7\" (UniqueName: \"kubernetes.io/projected/ea9c808c-62b4-4dc7-9761-bf6ee3700e34-kube-api-access-j7gv7\") pod \"console-operator-7cd56b8dbc-qbz82\" (UID: \"ea9c808c-62b4-4dc7-9761-bf6ee3700e34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:06:55.888071 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.888007    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-run-systemd\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.888071 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.888033    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7fw6\" (UniqueName: \"kubernetes.io/projected/4049e53b-bffe-40fb-beef-22acc171315a-kube-api-access-c7fw6\") pod \"network-metrics-daemon-fvsmv\" (UID: \"4049e53b-bffe-40fb-beef-22acc171315a\") " pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:55.888280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.888069    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/052e2399-f78d-4eb2-9da7-e2b098f86da7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:55.888280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.888102    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-multus-daemon-config\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.888280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.888136    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-registration-dir\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:55.888280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.888171    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45sk7\" (UniqueName: \"kubernetes.io/projected/ccf49970-fa36-4188-ab63-e14e440b73ec-kube-api-access-45sk7\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.888280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.888201    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e297483a-0b86-4481-9597-0c89e9a6a7ac-image-registry-private-configuration\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:55.888280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.888259    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-stats-auth\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:55.888558 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.888309    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bdc6308d-a259-45db-90ea-094a8d1dd3a4-cnibin\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:55.888558 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.888342    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vqtw4\" (UID: \"fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"
Oct 30 22:06:55.888558 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.888369    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/965e7b7f-9044-4c69-be4d-06ef9a1044e2-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:55.888558 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.888403    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-lib-modules\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.888558 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.888435    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk8vc\" (UniqueName: \"kubernetes.io/projected/e0c1f2fe-ca1b-4556-8df1-134c67cee4f8-kube-api-access-lk8vc\") pod \"ingress-canary-p89nn\" (UID: \"e0c1f2fe-ca1b-4556-8df1-134c67cee4f8\") " pod="openshift-ingress-canary/ingress-canary-p89nn"
Oct 30 22:06:55.888558 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.888465    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-kubelet\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.888558 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.888497    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-node-log\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.888558 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.888527    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c31a736c-cd3e-42be-b783-fc25436b5302-metrics-tls\") pod \"dns-default-t2gq6\" (UID: \"c31a736c-cd3e-42be-b783-fc25436b5302\") " pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:55.888914 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.888560    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e297483a-0b86-4481-9597-0c89e9a6a7ac-trusted-ca\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:55.888914 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.888590    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-run\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.888914 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.888619    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-cni-netd\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.888914 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.888647    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/47dc8b20-b520-4f4d-bd48-74b78ba300ef-env-overrides\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.888914 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.888680    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr8dq\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-kube-api-access-vr8dq\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:55.888914 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.888708    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv2fd\" (UniqueName: \"kubernetes.io/projected/c31a736c-cd3e-42be-b783-fc25436b5302-kube-api-access-vv2fd\") pod \"dns-default-t2gq6\" (UID: \"c31a736c-cd3e-42be-b783-fc25436b5302\") " pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:55.888914 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.888734    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-multus-daemon-config\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.888741    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-certificates\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.889438    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bdc6308d-a259-45db-90ea-094a8d1dd3a4-cni-binary-copy\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.889488    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0c1f2fe-ca1b-4556-8df1-134c67cee4f8-cert\") pod \"ingress-canary-p89nn\" (UID: \"e0c1f2fe-ca1b-4556-8df1-134c67cee4f8\") " pod="openshift-ingress-canary/ingress-canary-p89nn"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.889522    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/801d7671-e004-4da3-898e-664eb2f1b5c3-iptables-alerter-script\") pod \"iptables-alerter-86vll\" (UID: \"801d7671-e004-4da3-898e-664eb2f1b5c3\") " pod="openshift-network-operator/iptables-alerter-86vll"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.889567    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-run-netns\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.889593    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e297483a-0b86-4481-9597-0c89e9a6a7ac-installation-pull-secrets\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.889640    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605c1b60-1fa5-4257-a588-4b2e0a78bcca-service-ca-bundle\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.889654    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/965e7b7f-9044-4c69-be4d-06ef9a1044e2-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.889671    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bdc6308d-a259-45db-90ea-094a8d1dd3a4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.889715    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-run-ovn\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.889762    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-var-lib-kubelet\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.890396    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e297483a-0b86-4481-9597-0c89e9a6a7ac-trusted-ca\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.890414    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/801d7671-e004-4da3-898e-664eb2f1b5c3-iptables-alerter-script\") pod \"iptables-alerter-86vll\" (UID: \"801d7671-e004-4da3-898e-664eb2f1b5c3\") " pod="openshift-network-operator/iptables-alerter-86vll"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.890497    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e297483a-0b86-4481-9597-0c89e9a6a7ac-ca-trust-extracted\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.890538    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-certificates\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.890571    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f14d2c99-6d70-44bf-9046-7cc1b84f1e7e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-48hjx\" (UID: \"f14d2c99-6d70-44bf-9046-7cc1b84f1e7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-48hjx"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.890618    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qc4bw\" (UID: \"200052bd-2f0a-46d2-8b9b-6bd250b17b05\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.890655    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-etc-kubernetes\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.890760    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/47dc8b20-b520-4f4d-bd48-74b78ba300ef-env-overrides\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.890818    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e444015-5346-4106-93aa-ef43eb9d0be6-kubelet-config\") pod \"global-pull-secret-syncer-ljnhp\" (UID: \"1e444015-5346-4106-93aa-ef43eb9d0be6\") " pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.890855    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-tuned\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.890871    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-certificates\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.890883    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/47dc8b20-b520-4f4d-bd48-74b78ba300ef-ovn-node-metrics-cert\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.890946    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lztwd\" (UniqueName: \"kubernetes.io/projected/bdc6308d-a259-45db-90ea-094a8d1dd3a4-kube-api-access-lztwd\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.891003    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-stats-auth\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.891051    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e297483a-0b86-4481-9597-0c89e9a6a7ac-ca-trust-extracted\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.891127    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vqtw4\" (UID: \"fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.891210    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-tuned\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.891216    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-certificates\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.891274    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-modprobe-d\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.891316    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/47dc8b20-b520-4f4d-bd48-74b78ba300ef-ovnkube-script-lib\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.891349    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-bound-sa-token\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.891423    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/052e2399-f78d-4eb2-9da7-e2b098f86da7-data-volume\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.891456    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfdrz\" (UniqueName: \"kubernetes.io/projected/dd15a503-2539-4458-90cb-dae3e0e28e62-kube-api-access-kfdrz\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.891486    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/965e7b7f-9044-4c69-be4d-06ef9a1044e2-tmp\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.891513    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92nhs\" (UniqueName: \"kubernetes.io/projected/5b84639d-6b5d-4795-a356-8755ea234585-kube-api-access-92nhs\") pod \"service-ca-operator-544d749c8c-jd958\" (UID: \"5b84639d-6b5d-4795-a356-8755ea234585\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jd958"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.891547    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-cnibin\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.891557    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bdc6308d-a259-45db-90ea-094a8d1dd3a4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.891569    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bdc6308d-a259-45db-90ea-094a8d1dd3a4-cni-binary-copy\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.891582    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10d1cecc-9e30-4354-8b42-60ddceadff44-host\") pod \"node-ca-nwtzc\" (UID: \"10d1cecc-9e30-4354-8b42-60ddceadff44\") " pod="openshift-image-registry/node-ca-nwtzc"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.891687    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-sysctl-d\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.891726    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/052e2399-f78d-4eb2-9da7-e2b098f86da7-data-volume\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.891727    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-run-openvswitch\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.891777    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c31a736c-cd3e-42be-b783-fc25436b5302-tmp-dir\") pod \"dns-default-t2gq6\" (UID: \"c31a736c-cd3e-42be-b783-fc25436b5302\") " pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.891808    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-socket-dir\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.891833    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-device-dir\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.891872    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wthz4\" (UniqueName: \"kubernetes.io/projected/d303600a-2390-4ba9-8ea8-a58bd505a109-kube-api-access-wthz4\") pod \"network-check-target-zcl2w\" (UID: \"d303600a-2390-4ba9-8ea8-a58bd505a109\") " pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.891901    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swswj\" (UniqueName: \"kubernetes.io/projected/47dc8b20-b520-4f4d-bd48-74b78ba300ef-kube-api-access-swswj\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.891984    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c31a736c-cd3e-42be-b783-fc25436b5302-tmp-dir\") pod \"dns-default-t2gq6\" (UID: \"c31a736c-cd3e-42be-b783-fc25436b5302\") " pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:55.891960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.891985    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f14d2c99-6d70-44bf-9046-7cc1b84f1e7e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-48hjx\" (UID: \"f14d2c99-6d70-44bf-9046-7cc1b84f1e7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-48hjx"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.892041    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5cps\" (UniqueName: \"kubernetes.io/projected/052e2399-f78d-4eb2-9da7-e2b098f86da7-kube-api-access-q5cps\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.892045    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/965e7b7f-9044-4c69-be4d-06ef9a1044e2-tmp\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.892081    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2ec19290-25c4-4b52-8173-22c14e7c9056-hosts-file\") pod \"node-resolver-9dqlz\" (UID: \"2ec19290-25c4-4b52-8173-22c14e7c9056\") " pod="openshift-dns/node-resolver-9dqlz"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.892110    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e444015-5346-4106-93aa-ef43eb9d0be6-dbus\") pod \"global-pull-secret-syncer-ljnhp\" (UID: \"1e444015-5346-4106-93aa-ef43eb9d0be6\") " pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.892293    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.892357    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/47dc8b20-b520-4f4d-bd48-74b78ba300ef-ovnkube-script-lib\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.892383    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f14d2c99-6d70-44bf-9046-7cc1b84f1e7e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-48hjx\" (UID: \"f14d2c99-6d70-44bf-9046-7cc1b84f1e7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-48hjx"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.892426    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4wvm\" (UniqueName: \"kubernetes.io/projected/10d1cecc-9e30-4354-8b42-60ddceadff44-kube-api-access-b4wvm\") pod \"node-ca-nwtzc\" (UID: \"10d1cecc-9e30-4354-8b42-60ddceadff44\") " pod="openshift-image-registry/node-ca-nwtzc"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.892534    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-etc-openvswitch\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.892971    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-run-ovn-kubernetes\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.893027    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.893094    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmhwk\" (UniqueName: \"kubernetes.io/projected/f14d2c99-6d70-44bf-9046-7cc1b84f1e7e-kube-api-access-tmhwk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-48hjx\" (UID: \"f14d2c99-6d70-44bf-9046-7cc1b84f1e7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-48hjx"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.893155    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5b84639d-6b5d-4795-a356-8755ea234585-config\") pod \"service-ca-operator-544d749c8c-jd958\" (UID: \"5b84639d-6b5d-4795-a356-8755ea234585\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jd958"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.893276    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-var-lib-openvswitch\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.893313    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/052e2399-f78d-4eb2-9da7-e2b098f86da7-crio-socket\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.893340    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-system-cni-dir\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.893370    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-cni-binary-copy\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.893401    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5b84639d-6b5d-4795-a356-8755ea234585-config\") pod \"service-ca-operator-544d749c8c-jd958\" (UID: \"5b84639d-6b5d-4795-a356-8755ea234585\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jd958"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.893410    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-run-multus-certs\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.893512    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8ltk\" (UniqueName: \"kubernetes.io/projected/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-kube-api-access-k8ltk\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.893557    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrm4r\" (UniqueName: \"kubernetes.io/projected/88fe4dfd-dcda-444d-871b-24ca3fb10d77-kube-api-access-xrm4r\") pod \"volume-data-source-validator-58c9f5f46d-jh5sc\" (UID: \"88fe4dfd-dcda-444d-871b-24ca3fb10d77\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jh5sc"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.893610    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-cni-binary-copy\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.893625    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.893728    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-run-k8s-cni-cncf-io\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.893773    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ea4b0705-2096-41da-b78b-b62590f236e8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-n42rm\" (UID: \"ea4b0705-2096-41da-b78b-b62590f236e8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-n42rm"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.893807    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vqtw4\" (UID: \"fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.893832    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-sysctl-conf\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.893851    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9d1eecff-b22f-4864-a121-9fb507e783f2-agent-certs\") pod \"konnectivity-agent-2fhd5\" (UID: \"9d1eecff-b22f-4864-a121-9fb507e783f2\") " pod="kube-system/konnectivity-agent-2fhd5"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.893878    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea9c808c-62b4-4dc7-9761-bf6ee3700e34-serving-cert\") pod \"console-operator-7cd56b8dbc-qbz82\" (UID: \"ea9c808c-62b4-4dc7-9761-bf6ee3700e34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.893920    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.893961    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ea4b0705-2096-41da-b78b-b62590f236e8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-n42rm\" (UID: \"ea4b0705-2096-41da-b78b-b62590f236e8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-n42rm"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.893979    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-sysconfig\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.894006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894011    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec19290-25c4-4b52-8173-22c14e7c9056-tmp-dir\") pod \"node-resolver-9dqlz\" (UID: \"2ec19290-25c4-4b52-8173-22c14e7c9056\") " pod="openshift-dns/node-resolver-9dqlz"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894038    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr2ww\" (UniqueName: \"kubernetes.io/projected/605c1b60-1fa5-4257-a588-4b2e0a78bcca-kube-api-access-dr2ww\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894059    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894134    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-var-lib-cni-bin\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894161    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c31a736c-cd3e-42be-b783-fc25436b5302-config-volume\") pod \"dns-default-t2gq6\" (UID: \"c31a736c-cd3e-42be-b783-fc25436b5302\") " pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894209    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec19290-25c4-4b52-8173-22c14e7c9056-tmp-dir\") pod \"node-resolver-9dqlz\" (UID: \"2ec19290-25c4-4b52-8173-22c14e7c9056\") " pod="openshift-dns/node-resolver-9dqlz"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894252    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894292    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdc6308d-a259-45db-90ea-094a8d1dd3a4-system-cni-dir\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894326    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bptmv\" (UniqueName: \"kubernetes.io/projected/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-kube-api-access-bptmv\") pod \"cluster-samples-operator-6ddb6c8b94-vqtw4\" (UID: \"fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894365    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-host\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894432    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c31a736c-cd3e-42be-b783-fc25436b5302-config-volume\") pod \"dns-default-t2gq6\" (UID: \"c31a736c-cd3e-42be-b783-fc25436b5302\") " pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894537    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrsqc\" (UniqueName: \"kubernetes.io/projected/801d7671-e004-4da3-898e-664eb2f1b5c3-kube-api-access-jrsqc\") pod \"iptables-alerter-86vll\" (UID: \"801d7671-e004-4da3-898e-664eb2f1b5c3\") " pod="openshift-network-operator/iptables-alerter-86vll"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894558    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-multus-cni-dir\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894579    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-default-certificate\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894676    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/200052bd-2f0a-46d2-8b9b-6bd250b17b05-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qc4bw\" (UID: \"200052bd-2f0a-46d2-8b9b-6bd250b17b05\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894701    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-hostroot\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894719    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bdc6308d-a259-45db-90ea-094a8d1dd3a4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894771    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-default-certificate\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894813    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/801d7671-e004-4da3-898e-664eb2f1b5c3-host-slash\") pod \"iptables-alerter-86vll\" (UID: \"801d7671-e004-4da3-898e-664eb2f1b5c3\") " pod="openshift-network-operator/iptables-alerter-86vll"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894848    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6cjm\" (UniqueName: \"kubernetes.io/projected/200052bd-2f0a-46d2-8b9b-6bd250b17b05-kube-api-access-h6cjm\") pod \"cluster-monitoring-operator-884576c57-qc4bw\" (UID: \"200052bd-2f0a-46d2-8b9b-6bd250b17b05\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894881    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k4xc\" (UniqueName: \"kubernetes.io/projected/2ec19290-25c4-4b52-8173-22c14e7c9056-kube-api-access-5k4xc\") pod \"node-resolver-9dqlz\" (UID: \"2ec19290-25c4-4b52-8173-22c14e7c9056\") " pod="openshift-dns/node-resolver-9dqlz"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894882    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bdc6308d-a259-45db-90ea-094a8d1dd3a4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894900    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdqmz\" (UniqueName: \"kubernetes.io/projected/3f403b34-f155-49e6-b327-8ad89d00bbc3-kube-api-access-qdqmz\") pod \"kas-connection-checker-685895c7c9-k8lw5\" (UID: \"3f403b34-f155-49e6-b327-8ad89d00bbc3\") " pod="kube-system/kas-connection-checker-685895c7c9-k8lw5"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894901    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/200052bd-2f0a-46d2-8b9b-6bd250b17b05-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qc4bw\" (UID: \"200052bd-2f0a-46d2-8b9b-6bd250b17b05\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894916    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/965e7b7f-9044-4c69-be4d-06ef9a1044e2-snapshots\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894932    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9d1eecff-b22f-4864-a121-9fb507e783f2-konnectivity-ca\") pod \"konnectivity-agent-2fhd5\" (UID: \"9d1eecff-b22f-4864-a121-9fb507e783f2\") " pod="kube-system/konnectivity-agent-2fhd5"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894951    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-multus-conf-dir\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894974    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-bound-sa-token\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.894990    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcmwp\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-kube-api-access-vcmwp\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895012    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/965e7b7f-9044-4c69-be4d-06ef9a1044e2-snapshots\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895018    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-var-lib-cni-multus\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895079    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bdc6308d-a259-45db-90ea-094a8d1dd3a4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895144    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5b84639d-6b5d-4795-a356-8755ea234585-serving-cert\") pod \"service-ca-operator-544d749c8c-jd958\" (UID: \"5b84639d-6b5d-4795-a356-8755ea234585\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jd958"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895174    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b6651a05-7ba7-484f-8490-f22665bcb84f-installation-pull-secrets\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895190    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9d1eecff-b22f-4864-a121-9fb507e783f2-konnectivity-ca\") pod \"konnectivity-agent-2fhd5\" (UID: \"9d1eecff-b22f-4864-a121-9fb507e783f2\") " pod="kube-system/konnectivity-agent-2fhd5"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895249    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs\") pod \"network-metrics-daemon-fvsmv\" (UID: \"4049e53b-bffe-40fb-beef-22acc171315a\") " pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895272    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e444015-5346-4106-93aa-ef43eb9d0be6-original-pull-secret\") pod \"global-pull-secret-syncer-ljnhp\" (UID: \"1e444015-5346-4106-93aa-ef43eb9d0be6\") " pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895287    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/965e7b7f-9044-4c69-be4d-06ef9a1044e2-service-ca-bundle\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895310    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ea9c808c-62b4-4dc7-9761-bf6ee3700e34-trusted-ca\") pod \"console-operator-7cd56b8dbc-qbz82\" (UID: \"ea9c808c-62b4-4dc7-9761-bf6ee3700e34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:06:55.895306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895317    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5b84639d-6b5d-4795-a356-8755ea234585-serving-cert\") pod \"service-ca-operator-544d749c8c-jd958\" (UID: \"5b84639d-6b5d-4795-a356-8755ea234585\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jd958"
Oct 30 22:06:55.896646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895350    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/47dc8b20-b520-4f4d-bd48-74b78ba300ef-ovnkube-config\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.896646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895386    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/10d1cecc-9e30-4354-8b42-60ddceadff44-serviceca\") pod \"node-ca-nwtzc\" (UID: \"10d1cecc-9e30-4354-8b42-60ddceadff44\") " pod="openshift-image-registry/node-ca-nwtzc"
Oct 30 22:06:55.896646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895411    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qqhv\" (UniqueName: \"kubernetes.io/projected/965e7b7f-9044-4c69-be4d-06ef9a1044e2-kube-api-access-7qqhv\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:55.896646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895433    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/965e7b7f-9044-4c69-be4d-06ef9a1044e2-service-ca-bundle\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:55.896646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895439    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-etc-selinux\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:55.896646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895466    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ea9c808c-62b4-4dc7-9761-bf6ee3700e34-config\") pod \"console-operator-7cd56b8dbc-qbz82\" (UID: \"ea9c808c-62b4-4dc7-9761-bf6ee3700e34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:06:55.896646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895493    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-systemd-units\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.896646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895519    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-run-netns\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.896646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895548    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-log-socket\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.896646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895586    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/47dc8b20-b520-4f4d-bd48-74b78ba300ef-ovnkube-config\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.896646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895586    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/10d1cecc-9e30-4354-8b42-60ddceadff44-serviceca\") pod \"node-ca-nwtzc\" (UID: \"10d1cecc-9e30-4354-8b42-60ddceadff44\") " pod="openshift-image-registry/node-ca-nwtzc"
Oct 30 22:06:55.896646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895635    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea4b0705-2096-41da-b78b-b62590f236e8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-n42rm\" (UID: \"ea4b0705-2096-41da-b78b-b62590f236e8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-n42rm"
Oct 30 22:06:55.896646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895653    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ea9c808c-62b4-4dc7-9761-bf6ee3700e34-trusted-ca\") pod \"console-operator-7cd56b8dbc-qbz82\" (UID: \"ea9c808c-62b4-4dc7-9761-bf6ee3700e34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:06:55.896646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895662    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ea9c808c-62b4-4dc7-9761-bf6ee3700e34-config\") pod \"console-operator-7cd56b8dbc-qbz82\" (UID: \"ea9c808c-62b4-4dc7-9761-bf6ee3700e34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:06:55.896646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895674    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-sys\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.896646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895713    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-var-lib-kubelet\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.896646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895759    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-slash\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.896646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895787    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-multus-socket-dir-parent\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.896646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.895827    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-sys-fs\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:55.899548 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.899532    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 22:06:55.899629 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.899616    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 22:06:55.899857 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.899835    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 22:06:55.899969 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.899876    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 22:06:55.899969 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.899895    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 22:06:55.900150 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.900136    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c5qz2\""
Oct 30 22:06:55.900240 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.900202    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g5ww4\""
Oct 30 22:06:55.900441 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.900342    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 22:06:55.900441 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.900353    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 22:06:55.900441 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.900375    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 22:06:55.900441 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.900381    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 22:06:55.900441 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.900397    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tjmjw\""
Oct 30 22:06:55.900441 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.900392    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 22:06:55.900707 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.900481    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ggpvh\""
Oct 30 22:06:55.900707 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.900531    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 22:06:55.900707 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.900669    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 22:06:55.900707 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.900675    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8wxtm\""
Oct 30 22:06:55.900882 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.900716    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-44l7l\""
Oct 30 22:06:55.900882 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.900727    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nn9g8\""
Oct 30 22:06:55.900882 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.900736    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k6z4j\""
Oct 30 22:06:55.901075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.900895    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e297483a-0b86-4481-9597-0c89e9a6a7ac-image-registry-private-configuration\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:55.901075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.900935    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-26rrv\""
Oct 30 22:06:55.901075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.900950    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 22:06:55.901075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.900969    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z6gtt\""
Oct 30 22:06:55.901075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.901021    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7kw8g\""
Oct 30 22:06:55.901075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.901007    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 22:06:55.901313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.901081    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 22:06:55.901313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.901098    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xwfpf\""
Oct 30 22:06:55.901313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.901156    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-49l9d\""
Oct 30 22:06:55.901313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.901205    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-htsvp\""
Oct 30 22:06:55.901313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.901295    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 22:06:55.901453 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.901362    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pwrc4\""
Oct 30 22:06:55.901453 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.901411    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 22:06:55.901526 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.901454    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-bf7kf\""
Oct 30 22:06:55.901526 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.901510    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-rz55v\""
Oct 30 22:06:55.901586 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.901568    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e297483a-0b86-4481-9597-0c89e9a6a7ac-installation-pull-secrets\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:55.901721 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.901707    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f14d2c99-6d70-44bf-9046-7cc1b84f1e7e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-48hjx\" (UID: \"f14d2c99-6d70-44bf-9046-7cc1b84f1e7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-48hjx"
Oct 30 22:06:55.903465 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.903450    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/47dc8b20-b520-4f4d-bd48-74b78ba300ef-ovn-node-metrics-cert\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.904604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.904588    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea9c808c-62b4-4dc7-9761-bf6ee3700e34-serving-cert\") pod \"console-operator-7cd56b8dbc-qbz82\" (UID: \"ea9c808c-62b4-4dc7-9761-bf6ee3700e34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:06:55.904652 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.904640    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9d1eecff-b22f-4864-a121-9fb507e783f2-agent-certs\") pod \"konnectivity-agent-2fhd5\" (UID: \"9d1eecff-b22f-4864-a121-9fb507e783f2\") " pod="kube-system/konnectivity-agent-2fhd5"
Oct 30 22:06:55.905625 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.905602    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b6651a05-7ba7-484f-8490-f22665bcb84f-installation-pull-secrets\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:55.905738 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.905716    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e444015-5346-4106-93aa-ef43eb9d0be6-original-pull-secret\") pod \"global-pull-secret-syncer-ljnhp\" (UID: \"1e444015-5346-4106-93aa-ef43eb9d0be6\") " pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:55.906114 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.906094    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7wm8\" (UniqueName: \"kubernetes.io/projected/b8ee94be-1472-42e7-bf6e-d13e58671eeb-kube-api-access-h7wm8\") pod \"kas-connection-checker-685895c7c9-dvcrc\" (UID: \"b8ee94be-1472-42e7-bf6e-d13e58671eeb\") " pod="kube-system/kas-connection-checker-685895c7c9-dvcrc"
Oct 30 22:06:55.906318 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.906304    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h824x\" (UniqueName: \"kubernetes.io/projected/24c987a0-1cbf-4264-ba0e-586b738d08da-kube-api-access-h824x\") pod \"network-check-source-6f78d59f99-9xdrk\" (UID: \"24c987a0-1cbf-4264-ba0e-586b738d08da\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9xdrk"
Oct 30 22:06:55.906869 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.906853    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7gv7\" (UniqueName: \"kubernetes.io/projected/ea9c808c-62b4-4dc7-9761-bf6ee3700e34-kube-api-access-j7gv7\") pod \"console-operator-7cd56b8dbc-qbz82\" (UID: \"ea9c808c-62b4-4dc7-9761-bf6ee3700e34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:06:55.907750 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.907730    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b66td\" (UniqueName: \"kubernetes.io/projected/00b74f51-6c71-45f2-97af-6aea20b9eeaf-kube-api-access-b66td\") pod \"kas-connection-checker-685895c7c9-74sl2\" (UID: \"00b74f51-6c71-45f2-97af-6aea20b9eeaf\") " pod="kube-system/kas-connection-checker-685895c7c9-74sl2"
Oct 30 22:06:55.907845 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.907762    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv2fd\" (UniqueName: \"kubernetes.io/projected/c31a736c-cd3e-42be-b783-fc25436b5302-kube-api-access-vv2fd\") pod \"dns-default-t2gq6\" (UID: \"c31a736c-cd3e-42be-b783-fc25436b5302\") " pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:55.907845 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.907837    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/965e7b7f-9044-4c69-be4d-06ef9a1044e2-serving-cert\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:55.907925 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.907914    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b6651a05-7ba7-484f-8490-f22665bcb84f-image-registry-private-configuration\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:55.908371 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.908351    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr8dq\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-kube-api-access-vr8dq\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:55.910046 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.910027    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-bound-sa-token\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:55.910180 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.910163    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfdrz\" (UniqueName: \"kubernetes.io/projected/dd15a503-2539-4458-90cb-dae3e0e28e62-kube-api-access-kfdrz\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:55.910277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.910188    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92nhs\" (UniqueName: \"kubernetes.io/projected/5b84639d-6b5d-4795-a356-8755ea234585-kube-api-access-92nhs\") pod \"service-ca-operator-544d749c8c-jd958\" (UID: \"5b84639d-6b5d-4795-a356-8755ea234585\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jd958"
Oct 30 22:06:55.910277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.910193    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0c1f2fe-ca1b-4556-8df1-134c67cee4f8-cert\") pod \"ingress-canary-p89nn\" (UID: \"e0c1f2fe-ca1b-4556-8df1-134c67cee4f8\") " pod="openshift-ingress-canary/ingress-canary-p89nn"
Oct 30 22:06:55.910277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.910255    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45sk7\" (UniqueName: \"kubernetes.io/projected/ccf49970-fa36-4188-ab63-e14e440b73ec-kube-api-access-45sk7\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.910766 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.910750    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5cps\" (UniqueName: \"kubernetes.io/projected/052e2399-f78d-4eb2-9da7-e2b098f86da7-kube-api-access-q5cps\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:55.911277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.911251    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swswj\" (UniqueName: \"kubernetes.io/projected/47dc8b20-b520-4f4d-bd48-74b78ba300ef-kube-api-access-swswj\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.911698 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.911675    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4wvm\" (UniqueName: \"kubernetes.io/projected/10d1cecc-9e30-4354-8b42-60ddceadff44-kube-api-access-b4wvm\") pod \"node-ca-nwtzc\" (UID: \"10d1cecc-9e30-4354-8b42-60ddceadff44\") " pod="openshift-image-registry/node-ca-nwtzc"
Oct 30 22:06:55.911838 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.911821    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lztwd\" (UniqueName: \"kubernetes.io/projected/bdc6308d-a259-45db-90ea-094a8d1dd3a4-kube-api-access-lztwd\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:55.913024 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.913005    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8ltk\" (UniqueName: \"kubernetes.io/projected/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-kube-api-access-k8ltk\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.913470 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.913436    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrsqc\" (UniqueName: \"kubernetes.io/projected/801d7671-e004-4da3-898e-664eb2f1b5c3-kube-api-access-jrsqc\") pod \"iptables-alerter-86vll\" (UID: \"801d7671-e004-4da3-898e-664eb2f1b5c3\") " pod="openshift-network-operator/iptables-alerter-86vll"
Oct 30 22:06:55.913581 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.913566    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6cjm\" (UniqueName: \"kubernetes.io/projected/200052bd-2f0a-46d2-8b9b-6bd250b17b05-kube-api-access-h6cjm\") pod \"cluster-monitoring-operator-884576c57-qc4bw\" (UID: \"200052bd-2f0a-46d2-8b9b-6bd250b17b05\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"
Oct 30 22:06:55.913839 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.913822    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bptmv\" (UniqueName: \"kubernetes.io/projected/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-kube-api-access-bptmv\") pod \"cluster-samples-operator-6ddb6c8b94-vqtw4\" (UID: \"fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"
Oct 30 22:06:55.914122 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.914106    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrm4r\" (UniqueName: \"kubernetes.io/projected/88fe4dfd-dcda-444d-871b-24ca3fb10d77-kube-api-access-xrm4r\") pod \"volume-data-source-validator-58c9f5f46d-jh5sc\" (UID: \"88fe4dfd-dcda-444d-871b-24ca3fb10d77\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jh5sc"
Oct 30 22:06:55.914188 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.914164    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wthz4\" (UniqueName: \"kubernetes.io/projected/d303600a-2390-4ba9-8ea8-a58bd505a109-kube-api-access-wthz4\") pod \"network-check-target-zcl2w\" (UID: \"d303600a-2390-4ba9-8ea8-a58bd505a109\") " pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:55.914651 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.914632    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k4xc\" (UniqueName: \"kubernetes.io/projected/2ec19290-25c4-4b52-8173-22c14e7c9056-kube-api-access-5k4xc\") pod \"node-resolver-9dqlz\" (UID: \"2ec19290-25c4-4b52-8173-22c14e7c9056\") " pod="openshift-dns/node-resolver-9dqlz"
Oct 30 22:06:55.914925 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.914909    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcmwp\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-kube-api-access-vcmwp\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:55.915127 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.915113    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qqhv\" (UniqueName: \"kubernetes.io/projected/965e7b7f-9044-4c69-be4d-06ef9a1044e2-kube-api-access-7qqhv\") pod \"insights-operator-7c4986d449-vwshh\" (UID: \"965e7b7f-9044-4c69-be4d-06ef9a1044e2\") " pod="openshift-insights/insights-operator-7c4986d449-vwshh"
Oct 30 22:06:55.915302 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.915280    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr2ww\" (UniqueName: \"kubernetes.io/projected/605c1b60-1fa5-4257-a588-4b2e0a78bcca-kube-api-access-dr2ww\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:55.915622 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.915602    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmhwk\" (UniqueName: \"kubernetes.io/projected/f14d2c99-6d70-44bf-9046-7cc1b84f1e7e-kube-api-access-tmhwk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-48hjx\" (UID: \"f14d2c99-6d70-44bf-9046-7cc1b84f1e7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-48hjx"
Oct 30 22:06:55.915721 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.915677    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdqmz\" (UniqueName: \"kubernetes.io/projected/3f403b34-f155-49e6-b327-8ad89d00bbc3-kube-api-access-qdqmz\") pod \"kas-connection-checker-685895c7c9-k8lw5\" (UID: \"3f403b34-f155-49e6-b327-8ad89d00bbc3\") " pod="kube-system/kas-connection-checker-685895c7c9-k8lw5"
Oct 30 22:06:55.915795 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.915776    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-bound-sa-token\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:55.916479 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.916459    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7fw6\" (UniqueName: \"kubernetes.io/projected/4049e53b-bffe-40fb-beef-22acc171315a-kube-api-access-c7fw6\") pod \"network-metrics-daemon-fvsmv\" (UID: \"4049e53b-bffe-40fb-beef-22acc171315a\") " pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:55.916574 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.916502    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:55.923930 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.923912    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk8vc\" (UniqueName: \"kubernetes.io/projected/e0c1f2fe-ca1b-4556-8df1-134c67cee4f8-kube-api-access-lk8vc\") pod \"ingress-canary-p89nn\" (UID: \"e0c1f2fe-ca1b-4556-8df1-134c67cee4f8\") " pod="openshift-ingress-canary/ingress-canary-p89nn"
Oct 30 22:06:55.996630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.996601    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-systemd\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.996630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.996635    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-cni-bin\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.996818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.996653    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-kubernetes\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.996818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.996678    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-run-systemd\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.996818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.996701    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/052e2399-f78d-4eb2-9da7-e2b098f86da7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:55.996818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.996725    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-registration-dir\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:55.996818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.996729    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-systemd\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.996818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.996734    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-run-systemd\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.996818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.996752    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bdc6308d-a259-45db-90ea-094a8d1dd3a4-cnibin\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:55.996818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.996753    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-kubernetes\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.996818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.996773    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-lib-modules\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.996818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.996791    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-kubelet\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.996818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.996801    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-registration-dir\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:55.996818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.996810    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-node-log\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.996818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.996730    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-cni-bin\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.996818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.996810    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bdc6308d-a259-45db-90ea-094a8d1dd3a4-cnibin\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.996841    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-node-log\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.996861    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-kubelet\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.996907    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-lib-modules\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.996907    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c31a736c-cd3e-42be-b783-fc25436b5302-metrics-tls\") pod \"dns-default-t2gq6\" (UID: \"c31a736c-cd3e-42be-b783-fc25436b5302\") " pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.996958    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-run\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.996978    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-cni-netd\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:55.997000    4889 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997015    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-run-netns\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997032    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605c1b60-1fa5-4257-a588-4b2e0a78bcca-service-ca-bundle\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997035    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-cni-netd\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997059    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-run-ovn\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997063    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-run-netns\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.996999    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-run\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:55.997083    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c31a736c-cd3e-42be-b783-fc25436b5302-metrics-tls podName:c31a736c-cd3e-42be-b783-fc25436b5302 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:56.497063165 +0000 UTC m=+1.853429472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c31a736c-cd3e-42be-b783-fc25436b5302-metrics-tls") pod "dns-default-t2gq6" (UID: "c31a736c-cd3e-42be-b783-fc25436b5302") : secret "dns-default-metrics-tls" not found
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997090    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-run-ovn\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997116    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-var-lib-kubelet\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997151    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qc4bw\" (UID: \"200052bd-2f0a-46d2-8b9b-6bd250b17b05\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997154    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-var-lib-kubelet\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997178    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-etc-kubernetes\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997205    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e444015-5346-4106-93aa-ef43eb9d0be6-kubelet-config\") pod \"global-pull-secret-syncer-ljnhp\" (UID: \"1e444015-5346-4106-93aa-ef43eb9d0be6\") " pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997218    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-etc-kubernetes\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997255    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-modprobe-d\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:55.997268    4889 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997287    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-cnibin\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997313    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10d1cecc-9e30-4354-8b42-60ddceadff44-host\") pod \"node-ca-nwtzc\" (UID: \"10d1cecc-9e30-4354-8b42-60ddceadff44\") " pod="openshift-image-registry/node-ca-nwtzc"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997307    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e444015-5346-4106-93aa-ef43eb9d0be6-kubelet-config\") pod \"global-pull-secret-syncer-ljnhp\" (UID: \"1e444015-5346-4106-93aa-ef43eb9d0be6\") " pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997268    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/052e2399-f78d-4eb2-9da7-e2b098f86da7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997357    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-cnibin\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997360    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-sysctl-d\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:55.997382    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls podName:200052bd-2f0a-46d2-8b9b-6bd250b17b05 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:56.497366746 +0000 UTC m=+1.853733045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qc4bw" (UID: "200052bd-2f0a-46d2-8b9b-6bd250b17b05") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997387    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-modprobe-d\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997390    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10d1cecc-9e30-4354-8b42-60ddceadff44-host\") pod \"node-ca-nwtzc\" (UID: \"10d1cecc-9e30-4354-8b42-60ddceadff44\") " pod="openshift-image-registry/node-ca-nwtzc"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997433    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-run-openvswitch\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997463    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-socket-dir\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997469    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-sysctl-d\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997498    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-run-openvswitch\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997505    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-device-dir\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997518    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-socket-dir\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:55.997515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997540    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2ec19290-25c4-4b52-8173-22c14e7c9056-hosts-file\") pod \"node-resolver-9dqlz\" (UID: \"2ec19290-25c4-4b52-8173-22c14e7c9056\") " pod="openshift-dns/node-resolver-9dqlz"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997565    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-device-dir\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997588    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e444015-5346-4106-93aa-ef43eb9d0be6-dbus\") pod \"global-pull-secret-syncer-ljnhp\" (UID: \"1e444015-5346-4106-93aa-ef43eb9d0be6\") " pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997604    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997655    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e444015-5346-4106-93aa-ef43eb9d0be6-dbus\") pod \"global-pull-secret-syncer-ljnhp\" (UID: \"1e444015-5346-4106-93aa-ef43eb9d0be6\") " pod="kube-system/global-pull-secret-syncer-ljnhp"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997661    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605c1b60-1fa5-4257-a588-4b2e0a78bcca-service-ca-bundle\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997670    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2ec19290-25c4-4b52-8173-22c14e7c9056-hosts-file\") pod \"node-resolver-9dqlz\" (UID: \"2ec19290-25c4-4b52-8173-22c14e7c9056\") " pod="openshift-dns/node-resolver-9dqlz"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997672    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-etc-openvswitch\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:55.997692    4889 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:55.997705    4889 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54b978dcc8-sch2b: secret "image-registry-tls" not found
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997705    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-etc-openvswitch\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997733    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-run-ovn-kubernetes\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:55.997770    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls podName:e297483a-0b86-4481-9597-0c89e9a6a7ac nodeName:}" failed. No retries permitted until 2026-10-30 22:06:56.497755548 +0000 UTC m=+1.854121864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls") pod "image-registry-54b978dcc8-sch2b" (UID: "e297483a-0b86-4481-9597-0c89e9a6a7ac") : secret "image-registry-tls" not found
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997702    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-run-ovn-kubernetes\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997812    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997834    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-var-lib-openvswitch\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997851    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/052e2399-f78d-4eb2-9da7-e2b098f86da7-crio-socket\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997865    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-system-cni-dir\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997883    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-run-multus-certs\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997894    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-var-lib-openvswitch\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997935    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-system-cni-dir\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:55.997949    4889 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997955    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/052e2399-f78d-4eb2-9da7-e2b098f86da7-crio-socket\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997961    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:55.997961    4889 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b9d7c7dff-82x2r: secret "image-registry-tls" not found
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997980    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-run-k8s-cni-cncf-io\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997989    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-run-multus-certs\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.997996    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998000    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vqtw4\" (UID: \"fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:55.998011    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls podName:b6651a05-7ba7-484f-8490-f22665bcb84f nodeName:}" failed. No retries permitted until 2026-10-30 22:06:56.497997583 +0000 UTC m=+1.854363881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls") pod "image-registry-6b9d7c7dff-82x2r" (UID: "b6651a05-7ba7-484f-8490-f22665bcb84f") : secret "image-registry-tls" not found
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998027    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-run-k8s-cni-cncf-io\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998030    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-sysctl-conf\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:55.998064    4889 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998082    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998063    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998107    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-sysctl-conf\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:55.998112    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls podName:fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a nodeName:}" failed. No retries permitted until 2026-10-30 22:06:56.498094167 +0000 UTC m=+1.854460459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vqtw4" (UID: "fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a") : secret "samples-operator-tls" not found
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998134    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-sysconfig\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998151    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998166    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-var-lib-cni-bin\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998184    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998207    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdc6308d-a259-45db-90ea-094a8d1dd3a4-system-cni-dir\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998212    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-etc-sysconfig\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:55.998218    4889 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998244    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-host\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998266    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-multus-cni-dir\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998284    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-hostroot\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:55.998294    4889 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998309    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/801d7671-e004-4da3-898e-664eb2f1b5c3-host-slash\") pod \"iptables-alerter-86vll\" (UID: \"801d7671-e004-4da3-898e-664eb2f1b5c3\") " pod="openshift-network-operator/iptables-alerter-86vll"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998301    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-host\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:55.998339    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs podName:605c1b60-1fa5-4257-a588-4b2e0a78bcca nodeName:}" failed. No retries permitted until 2026-10-30 22:06:56.498325772 +0000 UTC m=+1.854692073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs") pod "router-default-fb457cdb7-m99g8" (UID: "605c1b60-1fa5-4257-a588-4b2e0a78bcca") : secret "router-metrics-certs-default" not found
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998345    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-hostroot\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998356    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-multus-conf-dir\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998368    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/801d7671-e004-4da3-898e-664eb2f1b5c3-host-slash\") pod \"iptables-alerter-86vll\" (UID: \"801d7671-e004-4da3-898e-664eb2f1b5c3\") " pod="openshift-network-operator/iptables-alerter-86vll"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998381    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-var-lib-cni-bin\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998385    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-multus-cni-dir\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:55.998402    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls podName:052e2399-f78d-4eb2-9da7-e2b098f86da7 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:56.498386383 +0000 UTC m=+1.854752681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mwjsl" (UID: "052e2399-f78d-4eb2-9da7-e2b098f86da7") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998407    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdc6308d-a259-45db-90ea-094a8d1dd3a4-system-cni-dir\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998386    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-var-lib-cni-multus\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998420    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-multus-conf-dir\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998427    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-host-var-lib-cni-multus\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998441    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bdc6308d-a259-45db-90ea-094a8d1dd3a4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998469    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs\") pod \"network-metrics-daemon-fvsmv\" (UID: \"4049e53b-bffe-40fb-beef-22acc171315a\") " pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998496    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-etc-selinux\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998538    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bdc6308d-a259-45db-90ea-094a8d1dd3a4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v69rv\" (UID: \"bdc6308d-a259-45db-90ea-094a8d1dd3a4\") " pod="openshift-multus/multus-additional-cni-plugins-v69rv"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998687    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-systemd-units\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998711    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-run-netns\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:55.998724    4889 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998737    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-log-socket\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998747    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-systemd-units\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998723    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-etc-selinux\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:55.998766    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs podName:4049e53b-bffe-40fb-beef-22acc171315a nodeName:}" failed. No retries permitted until 2026-10-30 22:06:56.498752702 +0000 UTC m=+1.855118998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs") pod "network-metrics-daemon-fvsmv" (UID: "4049e53b-bffe-40fb-beef-22acc171315a") : secret "metrics-daemon-secret" not found
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998759    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-run-netns\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998796    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-log-socket\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998799    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea4b0705-2096-41da-b78b-b62590f236e8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-n42rm\" (UID: \"ea4b0705-2096-41da-b78b-b62590f236e8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-n42rm"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998838    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-sys\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998871    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-var-lib-kubelet\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998896    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-slash\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998919    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-sys\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998918    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-multus-socket-dir-parent\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998932    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccf49970-fa36-4188-ab63-e14e440b73ec-var-lib-kubelet\") pod \"tuned-p4g27\" (UID: \"ccf49970-fa36-4188-ab63-e14e440b73ec\") " pod="openshift-cluster-node-tuning-operator/tuned-p4g27"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998956    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-sys-fs\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998959    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9-multus-socket-dir-parent\") pod \"multus-p8lsb\" (UID: \"4e64d9bd-0547-4c6d-adc4-8a3ce9bc0fb9\") " pod="openshift-multus/multus-p8lsb"
Oct 30 22:06:55.998962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.998980    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/47dc8b20-b520-4f4d-bd48-74b78ba300ef-host-slash\") pod \"ovnkube-node-8kfh5\" (UID: \"47dc8b20-b520-4f4d-bd48-74b78ba300ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:56.001202 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.999023    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd15a503-2539-4458-90cb-dae3e0e28e62-sys-fs\") pod \"aws-ebs-csi-driver-node-xlz8v\" (UID: \"dd15a503-2539-4458-90cb-dae3e0e28e62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlz8v"
Oct 30 22:06:56.001202 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:55.999117    4889 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 22:06:56.001496 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:56.001481    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea4b0705-2096-41da-b78b-b62590f236e8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-n42rm\" (UID: \"ea4b0705-2096-41da-b78b-b62590f236e8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-n42rm"
Oct 30 22:06:56.081493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:56.081460    4889 scope.go:122] "RemoveContainer" containerID="d60d0383fb24f739b8698d09dfc099d55168ef8507ac722ceb2547384d90c615"
Oct 30 22:06:56.082506 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:56.082489    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-n42rm"
Oct 30 22:06:56.097728 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:56.097689    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:56.257177 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:56.257143    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:56.408523 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:56.408501    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-n42rm"]
Oct 30 22:06:56.410435 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:56.410402    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea4b0705_2096_41da_b78b_b62590f236e8.slice/crio-1861843d884a22c5ed7cf6ebb5343301f89908c797fd5780dca4a88a2de7e992 WatchSource:0}: Error finding container 1861843d884a22c5ed7cf6ebb5343301f89908c797fd5780dca4a88a2de7e992: Status 404 returned error can't find the container with id 1861843d884a22c5ed7cf6ebb5343301f89908c797fd5780dca4a88a2de7e992
Oct 30 22:06:56.412590 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:56.412573    4889 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:06:56.433846 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:56.433817    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:56.504098 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:56.504023    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c31a736c-cd3e-42be-b783-fc25436b5302-metrics-tls\") pod \"dns-default-t2gq6\" (UID: \"c31a736c-cd3e-42be-b783-fc25436b5302\") " pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:56.504098 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:56.504063    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qc4bw\" (UID: \"200052bd-2f0a-46d2-8b9b-6bd250b17b05\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"
Oct 30 22:06:56.504098 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:56.504094    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:56.504373 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:56.504113    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:56.504373 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:56.504133    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vqtw4\" (UID: \"fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"
Oct 30 22:06:56.504373 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:56.504156    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:56.504373 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:56.504163    4889 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:06:56.504373 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:56.504180    4889 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:06:56.504373 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:56.504201    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:56.504373 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:56.504269    4889 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:56.504373 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:56.504270    4889 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:06:56.504373 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:56.504290    4889 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54b978dcc8-sch2b: secret "image-registry-tls" not found
Oct 30 22:06:56.504373 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:56.504303    4889 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:06:56.504373 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:56.504278    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c31a736c-cd3e-42be-b783-fc25436b5302-metrics-tls podName:c31a736c-cd3e-42be-b783-fc25436b5302 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:57.504250181 +0000 UTC m=+2.860616500 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c31a736c-cd3e-42be-b783-fc25436b5302-metrics-tls") pod "dns-default-t2gq6" (UID: "c31a736c-cd3e-42be-b783-fc25436b5302") : secret "dns-default-metrics-tls" not found
Oct 30 22:06:56.504373 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:56.504324    4889 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:06:56.504373 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:56.504336    4889 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:06:56.504373 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:56.504343    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls podName:200052bd-2f0a-46d2-8b9b-6bd250b17b05 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:57.504330388 +0000 UTC m=+2.860696683 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qc4bw" (UID: "200052bd-2f0a-46d2-8b9b-6bd250b17b05") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:06:56.504373 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:56.504349    4889 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b9d7c7dff-82x2r: secret "image-registry-tls" not found
Oct 30 22:06:56.504373 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:56.504361    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls podName:e297483a-0b86-4481-9597-0c89e9a6a7ac nodeName:}" failed. No retries permitted until 2026-10-30 22:06:57.504352036 +0000 UTC m=+2.860718334 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls") pod "image-registry-54b978dcc8-sch2b" (UID: "e297483a-0b86-4481-9597-0c89e9a6a7ac") : secret "image-registry-tls" not found
Oct 30 22:06:56.504373 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:56.504377    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls podName:052e2399-f78d-4eb2-9da7-e2b098f86da7 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:57.504369548 +0000 UTC m=+2.860735845 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mwjsl" (UID: "052e2399-f78d-4eb2-9da7-e2b098f86da7") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:56.504990 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:56.504393    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs podName:605c1b60-1fa5-4257-a588-4b2e0a78bcca nodeName:}" failed. No retries permitted until 2026-10-30 22:06:57.50438542 +0000 UTC m=+2.860751716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs") pod "router-default-fb457cdb7-m99g8" (UID: "605c1b60-1fa5-4257-a588-4b2e0a78bcca") : secret "router-metrics-certs-default" not found
Oct 30 22:06:56.504990 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:56.504412    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls podName:fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a nodeName:}" failed. No retries permitted until 2026-10-30 22:06:57.504402802 +0000 UTC m=+2.860769099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vqtw4" (UID: "fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a") : secret "samples-operator-tls" not found
Oct 30 22:06:56.504990 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:56.504428    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls podName:b6651a05-7ba7-484f-8490-f22665bcb84f nodeName:}" failed. No retries permitted until 2026-10-30 22:06:57.504420363 +0000 UTC m=+2.860786661 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls") pod "image-registry-6b9d7c7dff-82x2r" (UID: "b6651a05-7ba7-484f-8490-f22665bcb84f") : secret "image-registry-tls" not found
Oct 30 22:06:56.504990 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:56.504460    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs\") pod \"network-metrics-daemon-fvsmv\" (UID: \"4049e53b-bffe-40fb-beef-22acc171315a\") " pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:56.504990 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:56.504567    4889 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:06:56.504990 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:56.504607    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs podName:4049e53b-bffe-40fb-beef-22acc171315a nodeName:}" failed. No retries permitted until 2026-10-30 22:06:57.504597918 +0000 UTC m=+2.860964225 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs") pod "network-metrics-daemon-fvsmv" (UID: "4049e53b-bffe-40fb-beef-22acc171315a") : secret "metrics-daemon-secret" not found
Oct 30 22:06:56.591347 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:56.591311    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:56.770821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:56.770757    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:56.827872 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:56.827835    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-n42rm" event={"ID":"ea4b0705-2096-41da-b78b-b62590f236e8","Type":"ContainerStarted","Data":"1861843d884a22c5ed7cf6ebb5343301f89908c797fd5780dca4a88a2de7e992"}
Oct 30 22:06:56.829542 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:56.829518    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/2.log"
Oct 30 22:06:56.829915 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:56.829901    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/1.log"
Oct 30 22:06:56.829983 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:56.829932    4889 generic.go:356] "Generic (PLEG): container finished" podID="ea9c808c-62b4-4dc7-9761-bf6ee3700e34" containerID="80c4d71d82f4aacade456b78f25dd70472b6a3f8c1148b5d127e28d62f547d8e" exitCode=255
Oct 30 22:06:56.830043 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:56.830023    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82" event={"ID":"ea9c808c-62b4-4dc7-9761-bf6ee3700e34","Type":"ContainerDied","Data":"80c4d71d82f4aacade456b78f25dd70472b6a3f8c1148b5d127e28d62f547d8e"}
Oct 30 22:06:56.830094 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:56.830084    4889 scope.go:122] "RemoveContainer" containerID="d60d0383fb24f739b8698d09dfc099d55168ef8507ac722ceb2547384d90c615"
Oct 30 22:06:56.830796 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:56.830778    4889 scope.go:122] "RemoveContainer" containerID="80c4d71d82f4aacade456b78f25dd70472b6a3f8c1148b5d127e28d62f547d8e"
Oct 30 22:06:56.830947 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:56.830928    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qbz82_openshift-console-operator(ea9c808c-62b4-4dc7-9761-bf6ee3700e34)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82" podUID="ea9c808c-62b4-4dc7-9761-bf6ee3700e34"
Oct 30 22:06:57.119117 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:57.119042    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:57.515092 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:57.515058    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c31a736c-cd3e-42be-b783-fc25436b5302-metrics-tls\") pod \"dns-default-t2gq6\" (UID: \"c31a736c-cd3e-42be-b783-fc25436b5302\") " pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:57.515291 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:57.515112    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qc4bw\" (UID: \"200052bd-2f0a-46d2-8b9b-6bd250b17b05\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"
Oct 30 22:06:57.515291 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:57.515157    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:57.515291 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:57.515184    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:57.515291 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:57.515215    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vqtw4\" (UID: \"fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"
Oct 30 22:06:57.515291 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:57.515270    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:57.515562 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:57.515295    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:57.515562 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:57.515298    4889 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:06:57.515562 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:57.515309    4889 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:06:57.515562 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:57.515329    4889 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b9d7c7dff-82x2r: secret "image-registry-tls" not found
Oct 30 22:06:57.515562 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:57.515308    4889 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:06:57.515562 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:57.515362    4889 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:06:57.515562 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:57.515372    4889 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54b978dcc8-sch2b: secret "image-registry-tls" not found
Oct 30 22:06:57.515562 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:57.515369    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls podName:200052bd-2f0a-46d2-8b9b-6bd250b17b05 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:59.515350857 +0000 UTC m=+4.871717169 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qc4bw" (UID: "200052bd-2f0a-46d2-8b9b-6bd250b17b05") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:06:57.515562 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:57.515395    4889 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:06:57.515562 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:57.515397    4889 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:57.515562 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:57.515415    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls podName:b6651a05-7ba7-484f-8490-f22665bcb84f nodeName:}" failed. No retries permitted until 2026-10-30 22:06:59.515402081 +0000 UTC m=+4.871768373 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls") pod "image-registry-6b9d7c7dff-82x2r" (UID: "b6651a05-7ba7-484f-8490-f22665bcb84f") : secret "image-registry-tls" not found
Oct 30 22:06:57.515562 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:57.515432    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs\") pod \"network-metrics-daemon-fvsmv\" (UID: \"4049e53b-bffe-40fb-beef-22acc171315a\") " pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:57.515562 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:57.515444    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs podName:605c1b60-1fa5-4257-a588-4b2e0a78bcca nodeName:}" failed. No retries permitted until 2026-10-30 22:06:59.515428827 +0000 UTC m=+4.871795134 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs") pod "router-default-fb457cdb7-m99g8" (UID: "605c1b60-1fa5-4257-a588-4b2e0a78bcca") : secret "router-metrics-certs-default" not found
Oct 30 22:06:57.515562 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:57.515465    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls podName:fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a nodeName:}" failed. No retries permitted until 2026-10-30 22:06:59.515455971 +0000 UTC m=+4.871822275 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vqtw4" (UID: "fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a") : secret "samples-operator-tls" not found
Oct 30 22:06:57.515562 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:57.515483    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls podName:e297483a-0b86-4481-9597-0c89e9a6a7ac nodeName:}" failed. No retries permitted until 2026-10-30 22:06:59.515474633 +0000 UTC m=+4.871840941 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls") pod "image-registry-54b978dcc8-sch2b" (UID: "e297483a-0b86-4481-9597-0c89e9a6a7ac") : secret "image-registry-tls" not found
Oct 30 22:06:57.515562 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:57.515497    4889 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:06:57.515562 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:57.515501    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls podName:052e2399-f78d-4eb2-9da7-e2b098f86da7 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:59.51549237 +0000 UTC m=+4.871858680 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mwjsl" (UID: "052e2399-f78d-4eb2-9da7-e2b098f86da7") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:57.515562 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:57.515549    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs podName:4049e53b-bffe-40fb-beef-22acc171315a nodeName:}" failed. No retries permitted until 2026-10-30 22:06:59.515539229 +0000 UTC m=+4.871905536 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs") pod "network-metrics-daemon-fvsmv" (UID: "4049e53b-bffe-40fb-beef-22acc171315a") : secret "metrics-daemon-secret" not found
Oct 30 22:06:57.517698 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:57.517678    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c31a736c-cd3e-42be-b783-fc25436b5302-metrics-tls\") pod \"dns-default-t2gq6\" (UID: \"c31a736c-cd3e-42be-b783-fc25436b5302\") " pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:57.518037 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:57.518011    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9dqlz_2ec19290-25c4-4b52-8173-22c14e7c9056/dns-node-resolver/0.log"
Oct 30 22:06:57.518416 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:57.518393    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:57.590308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:57.590274    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:57.682309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:57.682214    4889 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:06:57.722901 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:57.722865    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:57.838738 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:57.838704    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-n42rm" event={"ID":"ea4b0705-2096-41da-b78b-b62590f236e8","Type":"ContainerStarted","Data":"afe2485e1b4a3e9685020e2f848a9c067ed39e080b70f5e211aa311f913458e1"}
Oct 30 22:06:57.840523 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:57.840503    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/2.log"
Oct 30 22:06:57.840826 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:57.840809    4889 scope.go:122] "RemoveContainer" containerID="80c4d71d82f4aacade456b78f25dd70472b6a3f8c1148b5d127e28d62f547d8e"
Oct 30 22:06:57.840998 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:57.840979    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qbz82_openshift-console-operator(ea9c808c-62b4-4dc7-9761-bf6ee3700e34)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82" podUID="ea9c808c-62b4-4dc7-9761-bf6ee3700e34"
Oct 30 22:06:57.883124 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:57.883098    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t2gq6"]
Oct 30 22:06:57.886236 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:06:57.886199    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31a736c_cd3e_42be_b783_fc25436b5302.slice/crio-8b502e2a4eb84b7ce259df070372af62f6955d92b497e00850d0707bfe9055c9 WatchSource:0}: Error finding container 8b502e2a4eb84b7ce259df070372af62f6955d92b497e00850d0707bfe9055c9: Status 404 returned error can't find the container with id 8b502e2a4eb84b7ce259df070372af62f6955d92b497e00850d0707bfe9055c9
Oct 30 22:06:57.921593 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:57.921559    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:58.089742 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:58.089711    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:58.291204 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:58.291168    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:06:58.322610 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:58.322571    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:58.717941 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:58.717914    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nwtzc_10d1cecc-9e30-4354-8b42-60ddceadff44/node-ca/0.log"
Oct 30 22:06:58.845553 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:58.845508    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t2gq6" event={"ID":"c31a736c-cd3e-42be-b783-fc25436b5302","Type":"ContainerStarted","Data":"8b502e2a4eb84b7ce259df070372af62f6955d92b497e00850d0707bfe9055c9"}
Oct 30 22:06:59.127832 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:59.127799    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:59.289563 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:59.289529    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:59.317721 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:59.317629    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-p89nn_e0c1f2fe-ca1b-4556-8df1-134c67cee4f8/serve-healthcheck-canary/0.log"
Oct 30 22:06:59.528870 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:59.528835    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:59.534242 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:59.534195    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:06:59.534373 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:59.534246    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:06:59.534373 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:59.534289    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vqtw4\" (UID: \"fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"
Oct 30 22:06:59.534373 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:59.534315    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:06:59.534373 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:59.534332    4889 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:06:59.534373 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:59.534350    4889 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54b978dcc8-sch2b: secret "image-registry-tls" not found
Oct 30 22:06:59.534373 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:59.534333    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:06:59.534655 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:59.534334    4889 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:06:59.534655 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:59.534400    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls podName:e297483a-0b86-4481-9597-0c89e9a6a7ac nodeName:}" failed. No retries permitted until 2026-10-30 22:07:03.534383795 +0000 UTC m=+8.890750088 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls") pod "image-registry-54b978dcc8-sch2b" (UID: "e297483a-0b86-4481-9597-0c89e9a6a7ac") : secret "image-registry-tls" not found
Oct 30 22:06:59.534655 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:59.534403    4889 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b9d7c7dff-82x2r: secret "image-registry-tls" not found
Oct 30 22:06:59.534655 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:59.534415    4889 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:06:59.534655 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:59.534416    4889 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:59.534655 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:59.534418    4889 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:06:59.534655 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:59.534447    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs\") pod \"network-metrics-daemon-fvsmv\" (UID: \"4049e53b-bffe-40fb-beef-22acc171315a\") " pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:06:59.534655 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:59.534455    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls podName:b6651a05-7ba7-484f-8490-f22665bcb84f nodeName:}" failed. No retries permitted until 2026-10-30 22:07:03.534439682 +0000 UTC m=+8.890805996 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls") pod "image-registry-6b9d7c7dff-82x2r" (UID: "b6651a05-7ba7-484f-8490-f22665bcb84f") : secret "image-registry-tls" not found
Oct 30 22:06:59.534655 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:59.534484    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls podName:052e2399-f78d-4eb2-9da7-e2b098f86da7 nodeName:}" failed. No retries permitted until 2026-10-30 22:07:03.534473823 +0000 UTC m=+8.890840118 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mwjsl" (UID: "052e2399-f78d-4eb2-9da7-e2b098f86da7") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:59.534655 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:59.534492    4889 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:06:59.534655 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:59.534501    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs podName:605c1b60-1fa5-4257-a588-4b2e0a78bcca nodeName:}" failed. No retries permitted until 2026-10-30 22:07:03.534491923 +0000 UTC m=+8.890858220 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs") pod "router-default-fb457cdb7-m99g8" (UID: "605c1b60-1fa5-4257-a588-4b2e0a78bcca") : secret "router-metrics-certs-default" not found
Oct 30 22:06:59.534655 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:59.534519    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls podName:fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a nodeName:}" failed. No retries permitted until 2026-10-30 22:07:03.534510424 +0000 UTC m=+8.890876730 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vqtw4" (UID: "fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a") : secret "samples-operator-tls" not found
Oct 30 22:06:59.534655 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:59.534534    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs podName:4049e53b-bffe-40fb-beef-22acc171315a nodeName:}" failed. No retries permitted until 2026-10-30 22:07:03.534527213 +0000 UTC m=+8.890893508 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs") pod "network-metrics-daemon-fvsmv" (UID: "4049e53b-bffe-40fb-beef-22acc171315a") : secret "metrics-daemon-secret" not found
Oct 30 22:06:59.534655 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:59.534607    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qc4bw\" (UID: \"200052bd-2f0a-46d2-8b9b-6bd250b17b05\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"
Oct 30 22:06:59.535403 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:59.534687    4889 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:06:59.535403 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:06:59.534717    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls podName:200052bd-2f0a-46d2-8b9b-6bd250b17b05 nodeName:}" failed. No retries permitted until 2026-10-30 22:07:03.534710421 +0000 UTC m=+8.891076768 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qc4bw" (UID: "200052bd-2f0a-46d2-8b9b-6bd250b17b05") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:06:59.717091 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:59.716998    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:59.822635 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:59.822608    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:06:59.822855 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:59.822840    4889 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:06:59.850400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:59.850374    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t2gq6" event={"ID":"c31a736c-cd3e-42be-b783-fc25436b5302","Type":"ContainerStarted","Data":"99cfc0b8395251b14c7d2ca2ac7c68a8bd5fb6c7ff4d92382962b1855f0fe5f4"}
Oct 30 22:06:59.850729 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:59.850419    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t2gq6" event={"ID":"c31a736c-cd3e-42be-b783-fc25436b5302","Type":"ContainerStarted","Data":"67775d6e60e7836c6293b0fae38da51004ae1f362a7f023c6db0d40b32f05d92"}
Oct 30 22:06:59.850729 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:59.850452    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:06:59.913003 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:06:59.912968    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:00.119689 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:00.119662    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-48hjx_f14d2c99-6d70-44bf-9046-7cc1b84f1e7e/kube-storage-version-migrator-operator/0.log"
Oct 30 22:07:00.130990 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:00.130959    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:00.495972 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:00.495943    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:01.128971 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:01.128939    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:01.303027 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:01.302991    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:01.494046 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:01.494019    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:07:01.494409 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:01.494396    4889 scope.go:122] "RemoveContainer" containerID="80c4d71d82f4aacade456b78f25dd70472b6a3f8c1148b5d127e28d62f547d8e"
Oct 30 22:07:01.494583 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:07:01.494566    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qbz82_openshift-console-operator(ea9c808c-62b4-4dc7-9761-bf6ee3700e34)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82" podUID="ea9c808c-62b4-4dc7-9761-bf6ee3700e34"
Oct 30 22:07:01.528023 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:01.527996    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:01.722677 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:01.722639    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:01.921278 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:01.921176    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:02.341682 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:02.341648    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:03.111888 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:03.111850    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:03.274873 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:03.274827    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:03.517200 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:03.517169    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:03.573317 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:03.573286    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs\") pod \"network-metrics-daemon-fvsmv\" (UID: \"4049e53b-bffe-40fb-beef-22acc171315a\") " pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:07:03.573410 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:03.573375    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qc4bw\" (UID: \"200052bd-2f0a-46d2-8b9b-6bd250b17b05\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"
Oct 30 22:07:03.573485 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:03.573407    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:07:03.573485 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:03.573435    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:07:03.573485 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:07:03.573453    4889 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:07:03.573623 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:03.573459    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vqtw4\" (UID: \"fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"
Oct 30 22:07:03.573623 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:07:03.573527    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs podName:4049e53b-bffe-40fb-beef-22acc171315a nodeName:}" failed. No retries permitted until 2026-10-30 22:07:11.573506805 +0000 UTC m=+16.929873115 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs") pod "network-metrics-daemon-fvsmv" (UID: "4049e53b-bffe-40fb-beef-22acc171315a") : secret "metrics-daemon-secret" not found
Oct 30 22:07:03.573623 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:07:03.573541    4889 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:07:03.573623 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:07:03.573595    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls podName:fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a nodeName:}" failed. No retries permitted until 2026-10-30 22:07:11.573580192 +0000 UTC m=+16.929946485 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vqtw4" (UID: "fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a") : secret "samples-operator-tls" not found
Oct 30 22:07:03.573623 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:03.573612    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:07:03.573876 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:03.573631    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:07:03.573876 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:07:03.573748    4889 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:07:03.573876 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:07:03.573827    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls podName:052e2399-f78d-4eb2-9da7-e2b098f86da7 nodeName:}" failed. No retries permitted until 2026-10-30 22:07:11.573811377 +0000 UTC m=+16.930177686 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mwjsl" (UID: "052e2399-f78d-4eb2-9da7-e2b098f86da7") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:07:03.576147 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:03.576117    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls\") pod \"image-registry-54b978dcc8-sch2b\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") " pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:07:03.576279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:03.576181    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls\") pod \"image-registry-6b9d7c7dff-82x2r\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") " pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:07:03.576328 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:03.576309    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/200052bd-2f0a-46d2-8b9b-6bd250b17b05-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qc4bw\" (UID: \"200052bd-2f0a-46d2-8b9b-6bd250b17b05\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"
Oct 30 22:07:03.576392 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:03.576379    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605c1b60-1fa5-4257-a588-4b2e0a78bcca-metrics-certs\") pod \"router-default-fb457cdb7-m99g8\" (UID: \"605c1b60-1fa5-4257-a588-4b2e0a78bcca\") " pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:07:03.580090 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:03.580075    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:07:03.581144 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:03.581130    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"
Oct 30 22:07:03.581765 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:03.581744    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:07:03.589284 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:03.589263    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:07:03.720170 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:03.720136    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:03.916796 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:03.916773    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw"]
Oct 30 22:07:03.919193 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:07:03.919100    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod200052bd_2f0a_46d2_8b9b_6bd250b17b05.slice/crio-ad2254a1f9d539548d458c6b35a276827a5754dddc48b53da17f4b5b02a6855d WatchSource:0}: Error finding container ad2254a1f9d539548d458c6b35a276827a5754dddc48b53da17f4b5b02a6855d: Status 404 returned error can't find the container with id ad2254a1f9d539548d458c6b35a276827a5754dddc48b53da17f4b5b02a6855d
Oct 30 22:07:03.923475 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:07:03.923443    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode297483a_0b86_4481_9597_0c89e9a6a7ac.slice/crio-960a9453fa1cfd87384c97465639c685b560344fbe56b075be39020cbe2d6bb7 WatchSource:0}: Error finding container 960a9453fa1cfd87384c97465639c685b560344fbe56b075be39020cbe2d6bb7: Status 404 returned error can't find the container with id 960a9453fa1cfd87384c97465639c685b560344fbe56b075be39020cbe2d6bb7
Oct 30 22:07:03.923625 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:03.923605    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:03.924172 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:07:03.924150    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod605c1b60_1fa5_4257_a588_4b2e0a78bcca.slice/crio-dbc0fe9b8691bd19052e7d0cc1d9157e7cb6b29bb5ed7da33f7f485dbd235875 WatchSource:0}: Error finding container dbc0fe9b8691bd19052e7d0cc1d9157e7cb6b29bb5ed7da33f7f485dbd235875: Status 404 returned error can't find the container with id dbc0fe9b8691bd19052e7d0cc1d9157e7cb6b29bb5ed7da33f7f485dbd235875
Oct 30 22:07:03.929105 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:03.929086    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-fb457cdb7-m99g8"]
Oct 30 22:07:03.932408 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:03.932386    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54b978dcc8-sch2b"]
Oct 30 22:07:03.937643 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:07:03.937618    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6651a05_7ba7_484f_8490_f22665bcb84f.slice/crio-2c81da34c69d09c245ca9555bab53d3da64e93c8e803bee9488b275699679dc6 WatchSource:0}: Error finding container 2c81da34c69d09c245ca9555bab53d3da64e93c8e803bee9488b275699679dc6: Status 404 returned error can't find the container with id 2c81da34c69d09c245ca9555bab53d3da64e93c8e803bee9488b275699679dc6
Oct 30 22:07:03.941929 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:03.941806    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b9d7c7dff-82x2r"]
Oct 30 22:07:04.326240 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:04.326193    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:04.495916 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:04.495885    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:04.868152 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:04.868108    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-fb457cdb7-m99g8" event={"ID":"605c1b60-1fa5-4257-a588-4b2e0a78bcca","Type":"ContainerStarted","Data":"6fec6ecab5bba30fc5e981a7491395c1b19a65334a2d024fe8a46c0c3b60f81f"}
Oct 30 22:07:04.868629 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:04.868176    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-fb457cdb7-m99g8" event={"ID":"605c1b60-1fa5-4257-a588-4b2e0a78bcca","Type":"ContainerStarted","Data":"dbc0fe9b8691bd19052e7d0cc1d9157e7cb6b29bb5ed7da33f7f485dbd235875"}
Oct 30 22:07:04.869862 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:04.869831    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54b978dcc8-sch2b" event={"ID":"e297483a-0b86-4481-9597-0c89e9a6a7ac","Type":"ContainerStarted","Data":"3114c799c8c16863fe0fbb1cbd582e22582d143ea754a72e392ed7bd704108b6"}
Oct 30 22:07:04.869976 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:04.869881    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54b978dcc8-sch2b" event={"ID":"e297483a-0b86-4481-9597-0c89e9a6a7ac","Type":"ContainerStarted","Data":"960a9453fa1cfd87384c97465639c685b560344fbe56b075be39020cbe2d6bb7"}
Oct 30 22:07:04.871153 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:04.871120    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw" event={"ID":"200052bd-2f0a-46d2-8b9b-6bd250b17b05","Type":"ContainerStarted","Data":"ad2254a1f9d539548d458c6b35a276827a5754dddc48b53da17f4b5b02a6855d"}
Oct 30 22:07:04.872568 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:04.872534    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r" event={"ID":"b6651a05-7ba7-484f-8490-f22665bcb84f","Type":"ContainerStarted","Data":"ce4ca0483412bd366b8c7de514163aff02302c4a6c31de2c526ac00d5716b9bf"}
Oct 30 22:07:04.872702 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:04.872605    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r" event={"ID":"b6651a05-7ba7-484f-8490-f22665bcb84f","Type":"ContainerStarted","Data":"2c81da34c69d09c245ca9555bab53d3da64e93c8e803bee9488b275699679dc6"}
Oct 30 22:07:05.119866 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:05.119772    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:05.518615 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:05.518558    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:05.580626 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:05.580575    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:07:05.582962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:05.582935    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:07:05.715406 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:05.715311    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:05.878119 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:05.878087    4889 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:07:05.878119 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:05.878100    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qc4bw" event={"ID":"200052bd-2f0a-46d2-8b9b-6bd250b17b05","Type":"ContainerStarted","Data":"b8459c8713a3b0ab9664e6f1d27a0639643a903d741b5d6d5dfaa957e7cc7b20"}
Oct 30 22:07:05.878567 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:05.878144    4889 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:07:05.935189 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:05.935159    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:06.096259 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:06.096195    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:06.261978 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:06.261946    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:06.365412 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:06.365335    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2fhd5"
Oct 30 22:07:06.472322 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:06.472286    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:06.645604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:06.645523    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:06.881394 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:06.881366    4889 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:07:07.618949 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:07.618908    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:07:07.766834 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:07.766800    4889 scope.go:122] "RemoveContainer" containerID="80c4d71d82f4aacade456b78f25dd70472b6a3f8c1148b5d127e28d62f547d8e"
Oct 30 22:07:07.886162 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:07.886138    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 22:07:07.886631 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:07.886616    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/2.log"
Oct 30 22:07:07.886678 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:07.886649    4889 generic.go:356] "Generic (PLEG): container finished" podID="ea9c808c-62b4-4dc7-9761-bf6ee3700e34" containerID="e30d625f253e79716bb248b463bc556aacbdb723d543f9e0eefab366182b2285" exitCode=255
Oct 30 22:07:07.886717 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:07.886707    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82" event={"ID":"ea9c808c-62b4-4dc7-9761-bf6ee3700e34","Type":"ContainerDied","Data":"e30d625f253e79716bb248b463bc556aacbdb723d543f9e0eefab366182b2285"}
Oct 30 22:07:07.886762 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:07.886752    4889 scope.go:122] "RemoveContainer" containerID="80c4d71d82f4aacade456b78f25dd70472b6a3f8c1148b5d127e28d62f547d8e"
Oct 30 22:07:07.887084 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:07.887061    4889 scope.go:122] "RemoveContainer" containerID="e30d625f253e79716bb248b463bc556aacbdb723d543f9e0eefab366182b2285"
Oct 30 22:07:07.887339 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:07:07.887317    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qbz82_openshift-console-operator(ea9c808c-62b4-4dc7-9761-bf6ee3700e34)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82" podUID="ea9c808c-62b4-4dc7-9761-bf6ee3700e34"
Oct 30 22:07:08.448820 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:08.448780    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:07:08.449039 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:08.448895    4889 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:07:08.450183 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:08.450161    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-fb457cdb7-m99g8"
Oct 30 22:07:08.895637 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:08.895609    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 22:07:08.928289 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:08.928263    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:07:09.855373 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:09.855345    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-t2gq6"
Oct 30 22:07:09.951323 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:09.951291    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54b978dcc8-sch2b"]
Oct 30 22:07:10.096501 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:10.096466    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:10.283769 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:10.283740    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:10.445308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:10.445274    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:11.494693 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:11.494666    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:07:11.495121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:11.495009    4889 scope.go:122] "RemoveContainer" containerID="e30d625f253e79716bb248b463bc556aacbdb723d543f9e0eefab366182b2285"
Oct 30 22:07:11.495211 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:07:11.495193    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qbz82_openshift-console-operator(ea9c808c-62b4-4dc7-9761-bf6ee3700e34)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82" podUID="ea9c808c-62b4-4dc7-9761-bf6ee3700e34"
Oct 30 22:07:11.647427 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:11.647372    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs\") pod \"network-metrics-daemon-fvsmv\" (UID: \"4049e53b-bffe-40fb-beef-22acc171315a\") " pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:07:11.647615 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:11.647499    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vqtw4\" (UID: \"fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"
Oct 30 22:07:11.647615 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:11.647532    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:07:11.650087 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:11.650060    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/052e2399-f78d-4eb2-9da7-e2b098f86da7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwjsl\" (UID: \"052e2399-f78d-4eb2-9da7-e2b098f86da7\") " pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:07:11.650087 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:11.650072    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4049e53b-bffe-40fb-beef-22acc171315a-metrics-certs\") pod \"network-metrics-daemon-fvsmv\" (UID: \"4049e53b-bffe-40fb-beef-22acc171315a\") " pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:07:11.650266 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:11.650104    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vqtw4\" (UID: \"fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"
Oct 30 22:07:11.680542 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:11.680521    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"
Oct 30 22:07:11.683680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:11.683650    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvsmv"
Oct 30 22:07:11.691616 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:11.691596    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mwjsl"
Oct 30 22:07:11.981805 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:11.981773    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fvsmv"]
Oct 30 22:07:11.985517 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:07:11.985489    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4049e53b_bffe_40fb_beef_22acc171315a.slice/crio-3fd12a24844645529c0f5756f1c6dd589b820b0aebbb39fb850c0967b75f3310 WatchSource:0}: Error finding container 3fd12a24844645529c0f5756f1c6dd589b820b0aebbb39fb850c0967b75f3310: Status 404 returned error can't find the container with id 3fd12a24844645529c0f5756f1c6dd589b820b0aebbb39fb850c0967b75f3310
Oct 30 22:07:11.994535 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:11.994515    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4"]
Oct 30 22:07:12.007190 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:12.007170    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mwjsl"]
Oct 30 22:07:12.609726 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:12.609661    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:12.796251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:12.796197    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:12.911978 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:12.911944    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mwjsl" event={"ID":"052e2399-f78d-4eb2-9da7-e2b098f86da7","Type":"ContainerStarted","Data":"f52bc71ee4fb001f8c6637ed94e3b7634a2dec1cb63d185b3a033dd70864a46d"}
Oct 30 22:07:12.912127 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:12.912005    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mwjsl" event={"ID":"052e2399-f78d-4eb2-9da7-e2b098f86da7","Type":"ContainerStarted","Data":"d21a0a640dae87eb80ced0fb1007b96c0e91253aabd30f5edfb0c5318baf6fd9"}
Oct 30 22:07:12.912127 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:12.912024    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mwjsl" event={"ID":"052e2399-f78d-4eb2-9da7-e2b098f86da7","Type":"ContainerStarted","Data":"49b4c24cc2c7216a415a1aa6959bcc3c8f925c12648929a1d653fa9b3def1dee"}
Oct 30 22:07:12.913570 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:12.913536    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fvsmv" event={"ID":"4049e53b-bffe-40fb-beef-22acc171315a","Type":"ContainerStarted","Data":"3fd12a24844645529c0f5756f1c6dd589b820b0aebbb39fb850c0967b75f3310"}
Oct 30 22:07:12.914950 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:12.914920    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4" event={"ID":"fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a","Type":"ContainerStarted","Data":"ba20d173e699569bf7e52b9d8fa0882469db9fddabc8eaf269dc189334fc3cc1"}
Oct 30 22:07:12.959899 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:12.959616    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:13.922452 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:13.922198    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fvsmv" event={"ID":"4049e53b-bffe-40fb-beef-22acc171315a","Type":"ContainerStarted","Data":"1bc664268a55c4fd37aa1b68cfc36caaee0ef4409ecb98a25694e82dcd836ab2"}
Oct 30 22:07:13.922452 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:13.922292    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fvsmv" event={"ID":"4049e53b-bffe-40fb-beef-22acc171315a","Type":"ContainerStarted","Data":"7653594a5d969ff950b83eaaae6cc2b90fab0fac3dceea5d8b06367676b71544"}
Oct 30 22:07:13.924954 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:13.924920    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4" event={"ID":"fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a","Type":"ContainerStarted","Data":"41b15791b36a5c060abc384ba30441f3d5c288a889760cd1ab7ba0b4afd98ca0"}
Oct 30 22:07:13.925094 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:13.924976    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqtw4" event={"ID":"fb3c8146-cb8b-4c73-a681-e3f48f8a6f5a","Type":"ContainerStarted","Data":"a9acbc7b7e3797e899aeb12caa772bb023f0971fc7779a3065c4e6c7f38dbc83"}
Oct 30 22:07:14.744533 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:14.744499    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8kfh5"
Oct 30 22:07:14.931019 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:14.930719    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mwjsl" event={"ID":"052e2399-f78d-4eb2-9da7-e2b098f86da7","Type":"ContainerStarted","Data":"74ea48666671a5f8f7819c736b8b4d35172bf441131687cf736574c7a2608e22"}
Oct 30 22:07:15.526364 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:15.526328    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:17.035055 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.035027    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:17.229171 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.229135    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:17.341695 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.341628    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4pfw9"]
Oct 30 22:07:17.346108 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.346091    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.372958 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.372926    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 22:07:17.373100 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.372973    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 22:07:17.375763 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.375747    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-btsx8\""
Oct 30 22:07:17.375953 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.375937    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 22:07:17.376020 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.375943    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 22:07:17.394670 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.394645    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a55933c9-03e1-4fac-9260-1869ca54070e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4pfw9\" (UID: \"a55933c9-03e1-4fac-9260-1869ca54070e\") " pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.394792 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.394683    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a55933c9-03e1-4fac-9260-1869ca54070e-node-exporter-accelerators-collector-config\") pod \"node-exporter-4pfw9\" (UID: \"a55933c9-03e1-4fac-9260-1869ca54070e\") " pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.394792 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.394708    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a55933c9-03e1-4fac-9260-1869ca54070e-node-exporter-textfile\") pod \"node-exporter-4pfw9\" (UID: \"a55933c9-03e1-4fac-9260-1869ca54070e\") " pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.394792 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.394770    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a55933c9-03e1-4fac-9260-1869ca54070e-root\") pod \"node-exporter-4pfw9\" (UID: \"a55933c9-03e1-4fac-9260-1869ca54070e\") " pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.394955 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.394836    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a55933c9-03e1-4fac-9260-1869ca54070e-sys\") pod \"node-exporter-4pfw9\" (UID: \"a55933c9-03e1-4fac-9260-1869ca54070e\") " pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.394955 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.394900    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a55933c9-03e1-4fac-9260-1869ca54070e-node-exporter-wtmp\") pod \"node-exporter-4pfw9\" (UID: \"a55933c9-03e1-4fac-9260-1869ca54070e\") " pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.394955 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.394927    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a55933c9-03e1-4fac-9260-1869ca54070e-metrics-client-ca\") pod \"node-exporter-4pfw9\" (UID: \"a55933c9-03e1-4fac-9260-1869ca54070e\") " pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.395052 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.394960    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a55933c9-03e1-4fac-9260-1869ca54070e-node-exporter-tls\") pod \"node-exporter-4pfw9\" (UID: \"a55933c9-03e1-4fac-9260-1869ca54070e\") " pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.395052 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.395000    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x95x8\" (UniqueName: \"kubernetes.io/projected/a55933c9-03e1-4fac-9260-1869ca54070e-kube-api-access-x95x8\") pod \"node-exporter-4pfw9\" (UID: \"a55933c9-03e1-4fac-9260-1869ca54070e\") " pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.410007 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.409982    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:17.495913 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.495885    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a55933c9-03e1-4fac-9260-1869ca54070e-root\") pod \"node-exporter-4pfw9\" (UID: \"a55933c9-03e1-4fac-9260-1869ca54070e\") " pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.495913 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.495917    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a55933c9-03e1-4fac-9260-1869ca54070e-sys\") pod \"node-exporter-4pfw9\" (UID: \"a55933c9-03e1-4fac-9260-1869ca54070e\") " pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.496121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.495941    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a55933c9-03e1-4fac-9260-1869ca54070e-node-exporter-wtmp\") pod \"node-exporter-4pfw9\" (UID: \"a55933c9-03e1-4fac-9260-1869ca54070e\") " pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.496121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.495959    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a55933c9-03e1-4fac-9260-1869ca54070e-metrics-client-ca\") pod \"node-exporter-4pfw9\" (UID: \"a55933c9-03e1-4fac-9260-1869ca54070e\") " pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.496121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.496017    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a55933c9-03e1-4fac-9260-1869ca54070e-sys\") pod \"node-exporter-4pfw9\" (UID: \"a55933c9-03e1-4fac-9260-1869ca54070e\") " pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.496121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.496044    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a55933c9-03e1-4fac-9260-1869ca54070e-node-exporter-tls\") pod \"node-exporter-4pfw9\" (UID: \"a55933c9-03e1-4fac-9260-1869ca54070e\") " pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.496121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.496078    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x95x8\" (UniqueName: \"kubernetes.io/projected/a55933c9-03e1-4fac-9260-1869ca54070e-kube-api-access-x95x8\") pod \"node-exporter-4pfw9\" (UID: \"a55933c9-03e1-4fac-9260-1869ca54070e\") " pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.496121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.496017    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a55933c9-03e1-4fac-9260-1869ca54070e-root\") pod \"node-exporter-4pfw9\" (UID: \"a55933c9-03e1-4fac-9260-1869ca54070e\") " pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.496121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.496107    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a55933c9-03e1-4fac-9260-1869ca54070e-node-exporter-wtmp\") pod \"node-exporter-4pfw9\" (UID: \"a55933c9-03e1-4fac-9260-1869ca54070e\") " pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.496511 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.496165    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a55933c9-03e1-4fac-9260-1869ca54070e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4pfw9\" (UID: \"a55933c9-03e1-4fac-9260-1869ca54070e\") " pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.496511 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.496243    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a55933c9-03e1-4fac-9260-1869ca54070e-node-exporter-accelerators-collector-config\") pod \"node-exporter-4pfw9\" (UID: \"a55933c9-03e1-4fac-9260-1869ca54070e\") " pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.496511 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.496292    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a55933c9-03e1-4fac-9260-1869ca54070e-node-exporter-textfile\") pod \"node-exporter-4pfw9\" (UID: \"a55933c9-03e1-4fac-9260-1869ca54070e\") " pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.496681 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.496621    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a55933c9-03e1-4fac-9260-1869ca54070e-node-exporter-textfile\") pod \"node-exporter-4pfw9\" (UID: \"a55933c9-03e1-4fac-9260-1869ca54070e\") " pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.496681 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.496630    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a55933c9-03e1-4fac-9260-1869ca54070e-metrics-client-ca\") pod \"node-exporter-4pfw9\" (UID: \"a55933c9-03e1-4fac-9260-1869ca54070e\") " pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.497110 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.497086    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a55933c9-03e1-4fac-9260-1869ca54070e-node-exporter-accelerators-collector-config\") pod \"node-exporter-4pfw9\" (UID: \"a55933c9-03e1-4fac-9260-1869ca54070e\") " pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.498714 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.498687    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a55933c9-03e1-4fac-9260-1869ca54070e-node-exporter-tls\") pod \"node-exporter-4pfw9\" (UID: \"a55933c9-03e1-4fac-9260-1869ca54070e\") " pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.498907 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.498892    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a55933c9-03e1-4fac-9260-1869ca54070e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4pfw9\" (UID: \"a55933c9-03e1-4fac-9260-1869ca54070e\") " pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.515555 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.515532    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x95x8\" (UniqueName: \"kubernetes.io/projected/a55933c9-03e1-4fac-9260-1869ca54070e-kube-api-access-x95x8\") pod \"node-exporter-4pfw9\" (UID: \"a55933c9-03e1-4fac-9260-1869ca54070e\") " pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.576470 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.576437    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:17.661631 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.661552    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4pfw9"
Oct 30 22:07:17.673904 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:07:17.673873    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda55933c9_03e1_4fac_9260_1869ca54070e.slice/crio-b1ca8ad1507b1abb9fb960f84a935c32481c669a456886cb173a273f41228330 WatchSource:0}: Error finding container b1ca8ad1507b1abb9fb960f84a935c32481c669a456886cb173a273f41228330: Status 404 returned error can't find the container with id b1ca8ad1507b1abb9fb960f84a935c32481c669a456886cb173a273f41228330
Oct 30 22:07:17.682721 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.682698    4889 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:07:17.683093 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.683077    4889 scope.go:122] "RemoveContainer" containerID="e30d625f253e79716bb248b463bc556aacbdb723d543f9e0eefab366182b2285"
Oct 30 22:07:17.683311 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:07:17.683292    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qbz82_openshift-console-operator(ea9c808c-62b4-4dc7-9761-bf6ee3700e34)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82" podUID="ea9c808c-62b4-4dc7-9761-bf6ee3700e34"
Oct 30 22:07:17.749425 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.749394    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:17.920318 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.920218    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:17.943922 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:17.943886    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4pfw9" event={"ID":"a55933c9-03e1-4fac-9260-1869ca54070e","Type":"ContainerStarted","Data":"b1ca8ad1507b1abb9fb960f84a935c32481c669a456886cb173a273f41228330"}
Oct 30 22:07:18.529338 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:18.529300    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:18.706635 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:18.706600    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:18.912066 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:18.911992    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:18.932542 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:18.932518    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:07:19.284079 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:19.284042    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:19.506154 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:19.506117    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:19.955359 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:19.955276    4889 generic.go:356] "Generic (PLEG): container finished" podID="a55933c9-03e1-4fac-9260-1869ca54070e" containerID="d93f811102f4ec6c81ccb212fe149f91474ea16cde82422987e07f3e7143b723" exitCode=0
Oct 30 22:07:19.955775 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:19.955358    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4pfw9" event={"ID":"a55933c9-03e1-4fac-9260-1869ca54070e","Type":"ContainerDied","Data":"d93f811102f4ec6c81ccb212fe149f91474ea16cde82422987e07f3e7143b723"}
Oct 30 22:07:19.956159 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:19.956069    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:07:20.591010 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:20.590964    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:20.751759 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:20.751720    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:20.954341 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:20.954243    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:20.961582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:20.961557    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4pfw9" event={"ID":"a55933c9-03e1-4fac-9260-1869ca54070e","Type":"ContainerStarted","Data":"2f3cc8bd1f10bed6206bc5e8b9e2b57281b7761f67dd6ec4953fdd104eabc5fb"}
Oct 30 22:07:20.961935 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:20.961606    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4pfw9" event={"ID":"a55933c9-03e1-4fac-9260-1869ca54070e","Type":"ContainerStarted","Data":"643a974ccc76410f560c85ea3c61875b201f7e74813e2e3fe32c68fc71050b7c"}
Oct 30 22:07:21.006798 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:21.006732    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4pfw9" podStartSLOduration=2.62506457 podStartE2EDuration="4.006717872s" totalImagesPullingTime="1.288263278s" totalInitContainerRuntime="93.390024ms" isStatefulPod=true podCreationTimestamp="2026-10-30 22:07:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:21.005519701 +0000 UTC m=+26.361886015" watchObservedRunningTime="2026-10-30 22:07:21.006717872 +0000 UTC m=+26.363084182"
Oct 30 22:07:22.158535 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:22.158474    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:22.363421 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:22.363380    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:24.423997 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:24.423963    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:26.425450 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:26.425399    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:26.593117 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:26.593074    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:27.713669 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:27.713634    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:27.882830 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:27.882782    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:28.111938 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:28.111904    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:28.287698 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:28.287664    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:28.294815 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:28.294792    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zcl2w"
Oct 30 22:07:28.767584 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:28.767546    4889 scope.go:122] "RemoveContainer" containerID="e30d625f253e79716bb248b463bc556aacbdb723d543f9e0eefab366182b2285"
Oct 30 22:07:28.994624 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:28.994601    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 22:07:28.994784 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:28.994694    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82" event={"ID":"ea9c808c-62b4-4dc7-9761-bf6ee3700e34","Type":"ContainerStarted","Data":"7a1f5719b7a5ab224e5b544b24568f86ebfaed3fcf7462ff7ebc954fd7f84f91"}
Oct 30 22:07:28.994981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:28.994957    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:07:29.271100 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:29.271073    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-qbz82"
Oct 30 22:07:29.618801 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:29.618724    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:29.873890 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:29.873815    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:30.040630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:30.040568    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:30.204841 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:30.204751    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:30.372062 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:30.372024    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:31.636611 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:31.636563    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:31.799699 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:31.799663    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:32.289534 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:32.289503    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b9d7c7dff-82x2r"]
Oct 30 22:07:32.442283 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:32.442251    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:32.604540 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:32.604453    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:32.767354 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:32.767320    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:34.973195 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:34.973093    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-54b978dcc8-sch2b" podUID="e297483a-0b86-4481-9597-0c89e9a6a7ac" containerName="registry" containerID="cri-o://3114c799c8c16863fe0fbb1cbd582e22582d143ea754a72e392ed7bd704108b6" gracePeriod=30
Oct 30 22:07:35.213972 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:35.213949    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:07:35.255071 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:35.255041    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e297483a-0b86-4481-9597-0c89e9a6a7ac-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e297483a-0b86-4481-9597-0c89e9a6a7ac-image-registry-private-configuration\") pod \"e297483a-0b86-4481-9597-0c89e9a6a7ac\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") "
Oct 30 22:07:35.255280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:35.255114    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e297483a-0b86-4481-9597-0c89e9a6a7ac-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e297483a-0b86-4481-9597-0c89e9a6a7ac-installation-pull-secrets\") pod \"e297483a-0b86-4481-9597-0c89e9a6a7ac\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") "
Oct 30 22:07:35.255280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:35.255151    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e297483a-0b86-4481-9597-0c89e9a6a7ac-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e297483a-0b86-4481-9597-0c89e9a6a7ac-ca-trust-extracted\") pod \"e297483a-0b86-4481-9597-0c89e9a6a7ac\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") "
Oct 30 22:07:35.255280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:35.255176    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-bound-sa-token\") pod \"e297483a-0b86-4481-9597-0c89e9a6a7ac\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") "
Oct 30 22:07:35.255280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:35.255213    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-kube-api-access-vcmwp\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-kube-api-access-vcmwp\") pod \"e297483a-0b86-4481-9597-0c89e9a6a7ac\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") "
Oct 30 22:07:35.255280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:35.255265    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-certificates\") pod \"e297483a-0b86-4481-9597-0c89e9a6a7ac\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") "
Oct 30 22:07:35.255554 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:35.255308    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls\") pod \"e297483a-0b86-4481-9597-0c89e9a6a7ac\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") "
Oct 30 22:07:35.255554 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:35.255348    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e297483a-0b86-4481-9597-0c89e9a6a7ac-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e297483a-0b86-4481-9597-0c89e9a6a7ac-trusted-ca\") pod \"e297483a-0b86-4481-9597-0c89e9a6a7ac\" (UID: \"e297483a-0b86-4481-9597-0c89e9a6a7ac\") "
Oct 30 22:07:35.255798 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:35.255735    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-certificates" pod "e297483a-0b86-4481-9597-0c89e9a6a7ac" (UID: "e297483a-0b86-4481-9597-0c89e9a6a7ac"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:35.255974 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:35.255949    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e297483a-0b86-4481-9597-0c89e9a6a7ac-trusted-ca" pod "e297483a-0b86-4481-9597-0c89e9a6a7ac" (UID: "e297483a-0b86-4481-9597-0c89e9a6a7ac"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:35.258044 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:35.258022    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e297483a-0b86-4481-9597-0c89e9a6a7ac-installation-pull-secrets" pod "e297483a-0b86-4481-9597-0c89e9a6a7ac" (UID: "e297483a-0b86-4481-9597-0c89e9a6a7ac"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:07:35.258198 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:35.258169    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls" pod "e297483a-0b86-4481-9597-0c89e9a6a7ac" (UID: "e297483a-0b86-4481-9597-0c89e9a6a7ac"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:35.258291 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:35.258272    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e297483a-0b86-4481-9597-0c89e9a6a7ac-image-registry-private-configuration" pod "e297483a-0b86-4481-9597-0c89e9a6a7ac" (UID: "e297483a-0b86-4481-9597-0c89e9a6a7ac"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:07:35.258410 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:35.258393    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-bound-sa-token" pod "e297483a-0b86-4481-9597-0c89e9a6a7ac" (UID: "e297483a-0b86-4481-9597-0c89e9a6a7ac"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:35.258486 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:35.258464    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-kube-api-access-vcmwp" pod "e297483a-0b86-4481-9597-0c89e9a6a7ac" (UID: "e297483a-0b86-4481-9597-0c89e9a6a7ac"). InnerVolumeSpecName "kube-api-access-vcmwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:35.264089 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:35.264060    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e297483a-0b86-4481-9597-0c89e9a6a7ac-ca-trust-extracted" pod "e297483a-0b86-4481-9597-0c89e9a6a7ac" (UID: "e297483a-0b86-4481-9597-0c89e9a6a7ac"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:35.356360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:35.356310    4889 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e297483a-0b86-4481-9597-0c89e9a6a7ac-trusted-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:07:35.356360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:35.356354    4889 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e297483a-0b86-4481-9597-0c89e9a6a7ac-image-registry-private-configuration\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:07:35.356360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:35.356366    4889 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e297483a-0b86-4481-9597-0c89e9a6a7ac-installation-pull-secrets\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:07:35.356360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:35.356376    4889 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e297483a-0b86-4481-9597-0c89e9a6a7ac-ca-trust-extracted\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:07:35.356360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:35.356384    4889 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-bound-sa-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:07:35.356676 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:35.356393    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcmwp\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-kube-api-access-vcmwp\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:07:35.356676 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:35.356401    4889 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-certificates\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:07:35.356676 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:35.356409    4889 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e297483a-0b86-4481-9597-0c89e9a6a7ac-registry-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:07:36.024046 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:36.024008    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54b978dcc8-sch2b" event={"ID":"e297483a-0b86-4481-9597-0c89e9a6a7ac","Type":"ContainerDied","Data":"3114c799c8c16863fe0fbb1cbd582e22582d143ea754a72e392ed7bd704108b6"}
Oct 30 22:07:36.024537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:36.023932    4889 generic.go:356] "Generic (PLEG): container finished" podID="e297483a-0b86-4481-9597-0c89e9a6a7ac" containerID="3114c799c8c16863fe0fbb1cbd582e22582d143ea754a72e392ed7bd704108b6" exitCode=0
Oct 30 22:07:36.024537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:36.024320    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54b978dcc8-sch2b" event={"ID":"e297483a-0b86-4481-9597-0c89e9a6a7ac","Type":"ContainerDied","Data":"960a9453fa1cfd87384c97465639c685b560344fbe56b075be39020cbe2d6bb7"}
Oct 30 22:07:36.024537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:36.024392    4889 scope.go:122] "RemoveContainer" containerID="3114c799c8c16863fe0fbb1cbd582e22582d143ea754a72e392ed7bd704108b6"
Oct 30 22:07:36.024537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:36.024476    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54b978dcc8-sch2b"
Oct 30 22:07:36.038019 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:36.038002    4889 scope.go:122] "RemoveContainer" containerID="3114c799c8c16863fe0fbb1cbd582e22582d143ea754a72e392ed7bd704108b6"
Oct 30 22:07:36.038299 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:07:36.038279    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3114c799c8c16863fe0fbb1cbd582e22582d143ea754a72e392ed7bd704108b6\": container with ID starting with 3114c799c8c16863fe0fbb1cbd582e22582d143ea754a72e392ed7bd704108b6 not found: ID does not exist" containerID="3114c799c8c16863fe0fbb1cbd582e22582d143ea754a72e392ed7bd704108b6"
Oct 30 22:07:36.038350 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:36.038309    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3114c799c8c16863fe0fbb1cbd582e22582d143ea754a72e392ed7bd704108b6"} err="failed to get container status \"3114c799c8c16863fe0fbb1cbd582e22582d143ea754a72e392ed7bd704108b6\": rpc error: code = NotFound desc = could not find container \"3114c799c8c16863fe0fbb1cbd582e22582d143ea754a72e392ed7bd704108b6\": container with ID starting with 3114c799c8c16863fe0fbb1cbd582e22582d143ea754a72e392ed7bd704108b6 not found: ID does not exist"
Oct 30 22:07:36.088309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:36.088283    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54b978dcc8-sch2b"]
Oct 30 22:07:36.100783 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:36.100752    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-54b978dcc8-sch2b"]
Oct 30 22:07:36.771906 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:36.771877    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e297483a-0b86-4481-9597-0c89e9a6a7ac" path="/var/lib/kubelet/pods/e297483a-0b86-4481-9597-0c89e9a6a7ac/volumes"
Oct 30 22:07:38.228713 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:38.228677    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:38.406359 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:38.406316    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:40.301999 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:07:40.301964    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:07:41.412411 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:41.412374    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:41.574112 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:41.574077    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:44.373785 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:44.373754    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:44.565089 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:44.565056    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:45.850841 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:45.850803    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:46.013238 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:46.013197    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:50.436903 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:50.436868    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:50.605482 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:50.605449    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:56.409750 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:56.409712    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:56.595801 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:56.595761    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:57.310043 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:57.309988    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r" podUID="b6651a05-7ba7-484f-8490-f22665bcb84f" containerName="registry" containerID="cri-o://ce4ca0483412bd366b8c7de514163aff02302c4a6c31de2c526ac00d5716b9bf" gracePeriod=30
Oct 30 22:07:57.547165 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:57.547140    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:07:57.635742 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:57.635664    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6651a05-7ba7-484f-8490-f22665bcb84f-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b6651a05-7ba7-484f-8490-f22665bcb84f-ca-trust-extracted\") pod \"b6651a05-7ba7-484f-8490-f22665bcb84f\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") "
Oct 30 22:07:57.635742 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:57.635721    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls\") pod \"b6651a05-7ba7-484f-8490-f22665bcb84f\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") "
Oct 30 22:07:57.635927 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:57.635760    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-kube-api-access-vr8dq\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-kube-api-access-vr8dq\") pod \"b6651a05-7ba7-484f-8490-f22665bcb84f\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") "
Oct 30 22:07:57.635927 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:57.635796    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6651a05-7ba7-484f-8490-f22665bcb84f-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b6651a05-7ba7-484f-8490-f22665bcb84f-installation-pull-secrets\") pod \"b6651a05-7ba7-484f-8490-f22665bcb84f\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") "
Oct 30 22:07:57.635927 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:57.635840    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6651a05-7ba7-484f-8490-f22665bcb84f-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6651a05-7ba7-484f-8490-f22665bcb84f-trusted-ca\") pod \"b6651a05-7ba7-484f-8490-f22665bcb84f\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") "
Oct 30 22:07:57.635927 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:57.635869    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-certificates\") pod \"b6651a05-7ba7-484f-8490-f22665bcb84f\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") "
Oct 30 22:07:57.635927 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:57.635899    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6651a05-7ba7-484f-8490-f22665bcb84f-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b6651a05-7ba7-484f-8490-f22665bcb84f-image-registry-private-configuration\") pod \"b6651a05-7ba7-484f-8490-f22665bcb84f\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") "
Oct 30 22:07:57.636157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:57.635932    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-bound-sa-token\") pod \"b6651a05-7ba7-484f-8490-f22665bcb84f\" (UID: \"b6651a05-7ba7-484f-8490-f22665bcb84f\") "
Oct 30 22:07:57.636416 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:57.636385    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6651a05-7ba7-484f-8490-f22665bcb84f-trusted-ca" pod "b6651a05-7ba7-484f-8490-f22665bcb84f" (UID: "b6651a05-7ba7-484f-8490-f22665bcb84f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:57.636672 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:57.636626    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-certificates" pod "b6651a05-7ba7-484f-8490-f22665bcb84f" (UID: "b6651a05-7ba7-484f-8490-f22665bcb84f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:57.638678 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:57.638656    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-bound-sa-token" pod "b6651a05-7ba7-484f-8490-f22665bcb84f" (UID: "b6651a05-7ba7-484f-8490-f22665bcb84f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:57.638877 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:57.638848    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-kube-api-access-vr8dq" pod "b6651a05-7ba7-484f-8490-f22665bcb84f" (UID: "b6651a05-7ba7-484f-8490-f22665bcb84f"). InnerVolumeSpecName "kube-api-access-vr8dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:57.638953 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:57.638881    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls" pod "b6651a05-7ba7-484f-8490-f22665bcb84f" (UID: "b6651a05-7ba7-484f-8490-f22665bcb84f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:57.638953 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:57.638912    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6651a05-7ba7-484f-8490-f22665bcb84f-installation-pull-secrets" pod "b6651a05-7ba7-484f-8490-f22665bcb84f" (UID: "b6651a05-7ba7-484f-8490-f22665bcb84f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:07:57.639068 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:57.638996    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6651a05-7ba7-484f-8490-f22665bcb84f-image-registry-private-configuration" pod "b6651a05-7ba7-484f-8490-f22665bcb84f" (UID: "b6651a05-7ba7-484f-8490-f22665bcb84f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:07:57.642387 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:57.642363    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6651a05-7ba7-484f-8490-f22665bcb84f-ca-trust-extracted" pod "b6651a05-7ba7-484f-8490-f22665bcb84f" (UID: "b6651a05-7ba7-484f-8490-f22665bcb84f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:57.737135 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:57.737101    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr8dq\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-kube-api-access-vr8dq\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:07:57.737135 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:57.737128    4889 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b6651a05-7ba7-484f-8490-f22665bcb84f-installation-pull-secrets\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:07:57.737135 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:57.737139    4889 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6651a05-7ba7-484f-8490-f22665bcb84f-trusted-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:07:57.737387 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:57.737148    4889 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-certificates\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:07:57.737387 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:57.737160    4889 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b6651a05-7ba7-484f-8490-f22665bcb84f-image-registry-private-configuration\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:07:57.737387 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:57.737168    4889 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-bound-sa-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:07:57.737387 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:57.737176    4889 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b6651a05-7ba7-484f-8490-f22665bcb84f-ca-trust-extracted\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:07:57.737387 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:57.737185    4889 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6651a05-7ba7-484f-8490-f22665bcb84f-registry-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:07:58.110353 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:58.110321    4889 generic.go:356] "Generic (PLEG): container finished" podID="b6651a05-7ba7-484f-8490-f22665bcb84f" containerID="ce4ca0483412bd366b8c7de514163aff02302c4a6c31de2c526ac00d5716b9bf" exitCode=0
Oct 30 22:07:58.110520 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:58.110364    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r"
Oct 30 22:07:58.110520 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:58.110362    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r" event={"ID":"b6651a05-7ba7-484f-8490-f22665bcb84f","Type":"ContainerDied","Data":"ce4ca0483412bd366b8c7de514163aff02302c4a6c31de2c526ac00d5716b9bf"}
Oct 30 22:07:58.110520 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:58.110503    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b9d7c7dff-82x2r" event={"ID":"b6651a05-7ba7-484f-8490-f22665bcb84f","Type":"ContainerDied","Data":"2c81da34c69d09c245ca9555bab53d3da64e93c8e803bee9488b275699679dc6"}
Oct 30 22:07:58.110620 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:58.110531    4889 scope.go:122] "RemoveContainer" containerID="ce4ca0483412bd366b8c7de514163aff02302c4a6c31de2c526ac00d5716b9bf"
Oct 30 22:07:58.119378 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:58.119356    4889 scope.go:122] "RemoveContainer" containerID="ce4ca0483412bd366b8c7de514163aff02302c4a6c31de2c526ac00d5716b9bf"
Oct 30 22:07:58.119650 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:07:58.119626    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce4ca0483412bd366b8c7de514163aff02302c4a6c31de2c526ac00d5716b9bf\": container with ID starting with ce4ca0483412bd366b8c7de514163aff02302c4a6c31de2c526ac00d5716b9bf not found: ID does not exist" containerID="ce4ca0483412bd366b8c7de514163aff02302c4a6c31de2c526ac00d5716b9bf"
Oct 30 22:07:58.119720 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:58.119658    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce4ca0483412bd366b8c7de514163aff02302c4a6c31de2c526ac00d5716b9bf"} err="failed to get container status \"ce4ca0483412bd366b8c7de514163aff02302c4a6c31de2c526ac00d5716b9bf\": rpc error: code = NotFound desc = could not find container \"ce4ca0483412bd366b8c7de514163aff02302c4a6c31de2c526ac00d5716b9bf\": container with ID starting with ce4ca0483412bd366b8c7de514163aff02302c4a6c31de2c526ac00d5716b9bf not found: ID does not exist"
Oct 30 22:07:58.161559 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:58.161532    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b9d7c7dff-82x2r"]
Oct 30 22:07:58.173913 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:58.173891    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6b9d7c7dff-82x2r"]
Oct 30 22:07:58.657113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:58.657082    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:58.771239 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:58.771197    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6651a05-7ba7-484f-8490-f22665bcb84f" path="/var/lib/kubelet/pods/b6651a05-7ba7-484f-8490-f22665bcb84f/volumes"
Oct 30 22:07:58.849075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:07:58.849045    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:00.119468 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:00.119437    4889 generic.go:356] "Generic (PLEG): container finished" podID="5b84639d-6b5d-4795-a356-8755ea234585" containerID="04afb2fe118c5022d23bd3a0737d555ce31d496ab0eabfe2c30ecf521f94ca48" exitCode=0
Oct 30 22:08:00.119859 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:00.119490    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jd958" event={"ID":"5b84639d-6b5d-4795-a356-8755ea234585","Type":"ContainerDied","Data":"04afb2fe118c5022d23bd3a0737d555ce31d496ab0eabfe2c30ecf521f94ca48"}
Oct 30 22:08:00.119859 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:00.119789    4889 scope.go:122] "RemoveContainer" containerID="04afb2fe118c5022d23bd3a0737d555ce31d496ab0eabfe2c30ecf521f94ca48"
Oct 30 22:08:00.251185 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:00.251153    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:00.416716 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:00.416624    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:01.125101 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:01.125065    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jd958" event={"ID":"5b84639d-6b5d-4795-a356-8755ea234585","Type":"ContainerStarted","Data":"f5077a9dd3cbdc9f4b8be91a4c39b75b9d63674c9b57d2254743733d3cf3bd55"}
Oct 30 22:08:04.023272 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:04.023216    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:07.767186 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:07.767148    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:07.928325 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:07.928285    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:10.299153 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:08:10.299119    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:08:11.411263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:11.411216    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:11.576304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:11.576267    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:20.197632 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:20.197598    4889 generic.go:356] "Generic (PLEG): container finished" podID="f14d2c99-6d70-44bf-9046-7cc1b84f1e7e" containerID="2d7e8d26fe3fd8f53c8f4bd67e9072719d373db10007cf1a6d23b1a0dedd9ac7" exitCode=0
Oct 30 22:08:20.198119 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:20.197672    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-48hjx" event={"ID":"f14d2c99-6d70-44bf-9046-7cc1b84f1e7e","Type":"ContainerDied","Data":"2d7e8d26fe3fd8f53c8f4bd67e9072719d373db10007cf1a6d23b1a0dedd9ac7"}
Oct 30 22:08:20.198119 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:20.198013    4889 scope.go:122] "RemoveContainer" containerID="2d7e8d26fe3fd8f53c8f4bd67e9072719d373db10007cf1a6d23b1a0dedd9ac7"
Oct 30 22:08:20.199269 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:20.199247    4889 generic.go:356] "Generic (PLEG): container finished" podID="965e7b7f-9044-4c69-be4d-06ef9a1044e2" containerID="afd146d373d52e0f98ce65ee914e9fbfa8ccd00e7769f91d073a424aafe5ef2f" exitCode=0
Oct 30 22:08:20.199401 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:20.199311    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-vwshh" event={"ID":"965e7b7f-9044-4c69-be4d-06ef9a1044e2","Type":"ContainerDied","Data":"afd146d373d52e0f98ce65ee914e9fbfa8ccd00e7769f91d073a424aafe5ef2f"}
Oct 30 22:08:20.199614 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:20.199599    4889 scope.go:122] "RemoveContainer" containerID="afd146d373d52e0f98ce65ee914e9fbfa8ccd00e7769f91d073a424aafe5ef2f"
Oct 30 22:08:21.052671 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:21.052638    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-d4j2d-zf2m9"]
Oct 30 22:08:21.052943 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:21.052930    4889 cpu_manager.go:414] "containerMap: removing container" podUID="b6651a05-7ba7-484f-8490-f22665bcb84f" containerName="registry"
Oct 30 22:08:21.052986 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:21.052945    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6651a05-7ba7-484f-8490-f22665bcb84f" containerName="registry"
Oct 30 22:08:21.052986 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:21.052956    4889 cpu_manager.go:414] "containerMap: removing container" podUID="e297483a-0b86-4481-9597-0c89e9a6a7ac" containerName="registry"
Oct 30 22:08:21.052986 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:21.052961    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e297483a-0b86-4481-9597-0c89e9a6a7ac" containerName="registry"
Oct 30 22:08:21.053079 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:21.053023    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6651a05-7ba7-484f-8490-f22665bcb84f" containerName="registry"
Oct 30 22:08:21.053079 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:21.053031    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="e297483a-0b86-4481-9597-0c89e9a6a7ac" containerName="registry"
Oct 30 22:08:21.055938 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:21.055918    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-d4j2d-zf2m9"
Oct 30 22:08:21.083293 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:21.083269    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vjfsq\""
Oct 30 22:08:21.084295 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:21.084276    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-d4j2d-zf2m9"]
Oct 30 22:08:21.111941 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:21.111914    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-trusted-ca-bundle\") pod \"periodic-gathering-d4j2d-zf2m9\" (UID: \"7c5c7fa5-4ac5-42d7-9492-6383f04393d0\") " pod="openshift-insights/periodic-gathering-d4j2d-zf2m9"
Oct 30 22:08:21.112094 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:21.111953    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bsv2\" (UniqueName: \"kubernetes.io/projected/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-kube-api-access-7bsv2\") pod \"periodic-gathering-d4j2d-zf2m9\" (UID: \"7c5c7fa5-4ac5-42d7-9492-6383f04393d0\") " pod="openshift-insights/periodic-gathering-d4j2d-zf2m9"
Oct 30 22:08:21.112094 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:21.111972    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-archives-path\") pod \"periodic-gathering-d4j2d-zf2m9\" (UID: \"7c5c7fa5-4ac5-42d7-9492-6383f04393d0\") " pod="openshift-insights/periodic-gathering-d4j2d-zf2m9"
Oct 30 22:08:21.112169 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:21.112098    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-service-ca-bundle\") pod \"periodic-gathering-d4j2d-zf2m9\" (UID: \"7c5c7fa5-4ac5-42d7-9492-6383f04393d0\") " pod="openshift-insights/periodic-gathering-d4j2d-zf2m9"
Oct 30 22:08:21.205135 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:21.205102    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-48hjx" event={"ID":"f14d2c99-6d70-44bf-9046-7cc1b84f1e7e","Type":"ContainerStarted","Data":"cc030d4e41ae78387fd87c69590e15817eadc7cb2a3924978f01c7eb4ac9d1a2"}
Oct 30 22:08:21.206996 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:21.206968    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-vwshh" event={"ID":"965e7b7f-9044-4c69-be4d-06ef9a1044e2","Type":"ContainerStarted","Data":"6fa9a7d21bd8ea8a85edeeb9b3cb6792d5729ac9a9b10a830e804726157bd28f"}
Oct 30 22:08:21.212595 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:21.212574    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-trusted-ca-bundle\") pod \"periodic-gathering-d4j2d-zf2m9\" (UID: \"7c5c7fa5-4ac5-42d7-9492-6383f04393d0\") " pod="openshift-insights/periodic-gathering-d4j2d-zf2m9"
Oct 30 22:08:21.212694 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:21.212667    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bsv2\" (UniqueName: \"kubernetes.io/projected/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-kube-api-access-7bsv2\") pod \"periodic-gathering-d4j2d-zf2m9\" (UID: \"7c5c7fa5-4ac5-42d7-9492-6383f04393d0\") " pod="openshift-insights/periodic-gathering-d4j2d-zf2m9"
Oct 30 22:08:21.212767 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:21.212736    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-archives-path\") pod \"periodic-gathering-d4j2d-zf2m9\" (UID: \"7c5c7fa5-4ac5-42d7-9492-6383f04393d0\") " pod="openshift-insights/periodic-gathering-d4j2d-zf2m9"
Oct 30 22:08:21.212848 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:21.212832    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-service-ca-bundle\") pod \"periodic-gathering-d4j2d-zf2m9\" (UID: \"7c5c7fa5-4ac5-42d7-9492-6383f04393d0\") " pod="openshift-insights/periodic-gathering-d4j2d-zf2m9"
Oct 30 22:08:21.213275 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:21.213250    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-archives-path\") pod \"periodic-gathering-d4j2d-zf2m9\" (UID: \"7c5c7fa5-4ac5-42d7-9492-6383f04393d0\") " pod="openshift-insights/periodic-gathering-d4j2d-zf2m9"
Oct 30 22:08:21.213487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:21.213465    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-service-ca-bundle\") pod \"periodic-gathering-d4j2d-zf2m9\" (UID: \"7c5c7fa5-4ac5-42d7-9492-6383f04393d0\") " pod="openshift-insights/periodic-gathering-d4j2d-zf2m9"
Oct 30 22:08:21.213664 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:21.213649    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-trusted-ca-bundle\") pod \"periodic-gathering-d4j2d-zf2m9\" (UID: \"7c5c7fa5-4ac5-42d7-9492-6383f04393d0\") " pod="openshift-insights/periodic-gathering-d4j2d-zf2m9"
Oct 30 22:08:21.232319 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:21.232291    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bsv2\" (UniqueName: \"kubernetes.io/projected/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-kube-api-access-7bsv2\") pod \"periodic-gathering-d4j2d-zf2m9\" (UID: \"7c5c7fa5-4ac5-42d7-9492-6383f04393d0\") " pod="openshift-insights/periodic-gathering-d4j2d-zf2m9"
Oct 30 22:08:21.367245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:21.367134    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-d4j2d-zf2m9"
Oct 30 22:08:21.660743 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:21.660718    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-d4j2d-zf2m9"]
Oct 30 22:08:22.212838 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:22.212805    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-d4j2d-zf2m9" event={"ID":"7c5c7fa5-4ac5-42d7-9492-6383f04393d0","Type":"ContainerStarted","Data":"eb12275c3c3f350900cbd73abda17691fce60f9541d4b5cf233f50a85dd815dd"}
Oct 30 22:08:22.213372 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:22.212853    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-d4j2d-zf2m9" event={"ID":"7c5c7fa5-4ac5-42d7-9492-6383f04393d0","Type":"ContainerStarted","Data":"f7b03db6ceeef052052dbcef8bba8552c7704eaf1b628f48dd24e1db615e80d2"}
Oct 30 22:08:22.252996 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:22.252944    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-d4j2d-zf2m9" podStartSLOduration=1.252929111 podStartE2EDuration="1.252929111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:22.251986293 +0000 UTC m=+87.608352609" watchObservedRunningTime="2026-10-30 22:08:22.252929111 +0000 UTC m=+87.609295426"
Oct 30 22:08:34.148132 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:34.148093    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:34.314056 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:34.314009    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:40.302040 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:08:40.301990    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:08:44.502807 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:44.502770    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:50.685591 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:50.685555    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:50.858015 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:50.857959    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:56.414350 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:56.414309    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:56.580539 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:08:56.580504    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:10.301033 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:09:10.300952    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:09:29.836915 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:09:29.836885    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:30.003958 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:09:30.003917    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:31.497606 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:09:31.497564    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:31.683122 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:09:31.683091    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:40.301373 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:09:40.301337    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:09:44.598878 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:09:44.598843    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:46.519784 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:09:46.519750    4889 generic.go:356] "Generic (PLEG): container finished" podID="7c5c7fa5-4ac5-42d7-9492-6383f04393d0" containerID="eb12275c3c3f350900cbd73abda17691fce60f9541d4b5cf233f50a85dd815dd" exitCode=0
Oct 30 22:09:46.520183 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:09:46.519798    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-d4j2d-zf2m9" event={"ID":"7c5c7fa5-4ac5-42d7-9492-6383f04393d0","Type":"ContainerDied","Data":"eb12275c3c3f350900cbd73abda17691fce60f9541d4b5cf233f50a85dd815dd"}
Oct 30 22:09:47.651389 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:09:47.651367    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-d4j2d-zf2m9"
Oct 30 22:09:47.663458 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:09:47.663435    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-trusted-ca-bundle\") pod \"7c5c7fa5-4ac5-42d7-9492-6383f04393d0\" (UID: \"7c5c7fa5-4ac5-42d7-9492-6383f04393d0\") "
Oct 30 22:09:47.663532 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:09:47.663478    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-service-ca-bundle\") pod \"7c5c7fa5-4ac5-42d7-9492-6383f04393d0\" (UID: \"7c5c7fa5-4ac5-42d7-9492-6383f04393d0\") "
Oct 30 22:09:47.663532 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:09:47.663510    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-archives-path\") pod \"7c5c7fa5-4ac5-42d7-9492-6383f04393d0\" (UID: \"7c5c7fa5-4ac5-42d7-9492-6383f04393d0\") "
Oct 30 22:09:47.663810 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:09:47.663789    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-archives-path" pod "7c5c7fa5-4ac5-42d7-9492-6383f04393d0" (UID: "7c5c7fa5-4ac5-42d7-9492-6383f04393d0"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:47.663810 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:09:47.663799    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-service-ca-bundle" pod "7c5c7fa5-4ac5-42d7-9492-6383f04393d0" (UID: "7c5c7fa5-4ac5-42d7-9492-6383f04393d0"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:47.663893 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:09:47.663840    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-trusted-ca-bundle" pod "7c5c7fa5-4ac5-42d7-9492-6383f04393d0" (UID: "7c5c7fa5-4ac5-42d7-9492-6383f04393d0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:47.764431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:09:47.764395    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-kube-api-access-7bsv2\" (UniqueName: \"kubernetes.io/projected/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-kube-api-access-7bsv2\") pod \"7c5c7fa5-4ac5-42d7-9492-6383f04393d0\" (UID: \"7c5c7fa5-4ac5-42d7-9492-6383f04393d0\") "
Oct 30 22:09:47.764610 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:09:47.764588    4889 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-service-ca-bundle\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:09:47.764666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:09:47.764608    4889 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-archives-path\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:09:47.764666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:09:47.764622    4889 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-trusted-ca-bundle\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:09:47.766762 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:09:47.766734    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-kube-api-access-7bsv2" pod "7c5c7fa5-4ac5-42d7-9492-6383f04393d0" (UID: "7c5c7fa5-4ac5-42d7-9492-6383f04393d0"). InnerVolumeSpecName "kube-api-access-7bsv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:47.865383 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:09:47.865296    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bsv2\" (UniqueName: \"kubernetes.io/projected/7c5c7fa5-4ac5-42d7-9492-6383f04393d0-kube-api-access-7bsv2\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:09:48.528390 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:09:48.528360    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-d4j2d-zf2m9"
Oct 30 22:09:48.528390 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:09:48.528370    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-d4j2d-zf2m9" event={"ID":"7c5c7fa5-4ac5-42d7-9492-6383f04393d0","Type":"ContainerDied","Data":"f7b03db6ceeef052052dbcef8bba8552c7704eaf1b628f48dd24e1db615e80d2"}
Oct 30 22:09:48.528587 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:09:48.528417    4889 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7b03db6ceeef052052dbcef8bba8552c7704eaf1b628f48dd24e1db615e80d2"
Oct 30 22:09:49.044434 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:09:49.044393    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:49.227071 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:09:49.227028    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:10.299716 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:10:10.299678    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:10:21.176856 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:21.176811    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:21.402566 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:21.402525    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:34.851715 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:34.851680    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:40.299431 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:10:40.299391    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:10:44.720808 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:44.720775    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:58.385299 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:58.385259    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:58.577506 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:58.577468    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:59.467960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.467926    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p"]
Oct 30 22:10:59.468358 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.468278    4889 cpu_manager.go:414] "containerMap: removing container" podUID="7c5c7fa5-4ac5-42d7-9492-6383f04393d0" containerName="insights-gathering"
Oct 30 22:10:59.468358 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.468289    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c5c7fa5-4ac5-42d7-9492-6383f04393d0" containerName="insights-gathering"
Oct 30 22:10:59.468358 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.468343    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c5c7fa5-4ac5-42d7-9492-6383f04393d0" containerName="insights-gathering"
Oct 30 22:10:59.471540 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.471520    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p"
Oct 30 22:10:59.490114 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.490087    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-qjg8k\""
Oct 30 22:10:59.502421 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.502392    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 22:10:59.502421 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.502419    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 22:10:59.503292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.503267    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p"]
Oct 30 22:10:59.585993 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.585957    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/711ee0fc-3e1a-40b5-9a71-724d6b515761-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p\" (UID: \"711ee0fc-3e1a-40b5-9a71-724d6b515761\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p"
Oct 30 22:10:59.586196 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.586010    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6jhz\" (UniqueName: \"kubernetes.io/projected/711ee0fc-3e1a-40b5-9a71-724d6b515761-kube-api-access-l6jhz\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p\" (UID: \"711ee0fc-3e1a-40b5-9a71-724d6b515761\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p"
Oct 30 22:10:59.586196 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.586036    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/711ee0fc-3e1a-40b5-9a71-724d6b515761-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p\" (UID: \"711ee0fc-3e1a-40b5-9a71-724d6b515761\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p"
Oct 30 22:10:59.686977 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.686940    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6jhz\" (UniqueName: \"kubernetes.io/projected/711ee0fc-3e1a-40b5-9a71-724d6b515761-kube-api-access-l6jhz\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p\" (UID: \"711ee0fc-3e1a-40b5-9a71-724d6b515761\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p"
Oct 30 22:10:59.686977 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.686978    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/711ee0fc-3e1a-40b5-9a71-724d6b515761-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p\" (UID: \"711ee0fc-3e1a-40b5-9a71-724d6b515761\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p"
Oct 30 22:10:59.687272 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.687021    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/711ee0fc-3e1a-40b5-9a71-724d6b515761-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p\" (UID: \"711ee0fc-3e1a-40b5-9a71-724d6b515761\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p"
Oct 30 22:10:59.687416 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.687398    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/711ee0fc-3e1a-40b5-9a71-724d6b515761-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p\" (UID: \"711ee0fc-3e1a-40b5-9a71-724d6b515761\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p"
Oct 30 22:10:59.687487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.687464    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/711ee0fc-3e1a-40b5-9a71-724d6b515761-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p\" (UID: \"711ee0fc-3e1a-40b5-9a71-724d6b515761\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p"
Oct 30 22:10:59.706952 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.706917    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6jhz\" (UniqueName: \"kubernetes.io/projected/711ee0fc-3e1a-40b5-9a71-724d6b515761-kube-api-access-l6jhz\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p\" (UID: \"711ee0fc-3e1a-40b5-9a71-724d6b515761\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p"
Oct 30 22:10:59.786981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.786955    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p"
Oct 30 22:10:59.901017 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.900986    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"]
Oct 30 22:10:59.904428 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.904407    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"
Oct 30 22:10:59.922118 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.922092    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 22:10:59.923600 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.923588    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 22:10:59.923657 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.923598    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 22:10:59.923703 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.923687    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ggg7r\""
Oct 30 22:10:59.923824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.923809    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 22:10:59.927961 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.927941    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"]
Oct 30 22:10:59.988964 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.988930    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:59.989784 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.989764    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-dlh79\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"
Oct 30 22:10:59.989843 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.989798    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-dlh79\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"
Oct 30 22:10:59.989843 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.989820    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-dlh79\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"
Oct 30 22:10:59.989920 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.989839    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-dlh79\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"
Oct 30 22:10:59.989969 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.989953    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/df96496e-4bf3-470b-b4d6-8bac68520ab7-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-dlh79\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"
Oct 30 22:10:59.990005 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.989986    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/df96496e-4bf3-470b-b4d6-8bac68520ab7-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-dlh79\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"
Oct 30 22:10:59.990037 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:10:59.990005    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnzz4\" (UniqueName: \"kubernetes.io/projected/df96496e-4bf3-470b-b4d6-8bac68520ab7-kube-api-access-fnzz4\") pod \"istiod-openshift-gateway-7876bfcb74-dlh79\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"
Oct 30 22:11:00.084474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:00.084451    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p"]
Oct 30 22:11:00.086205 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:11:00.086177    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711ee0fc_3e1a_40b5_9a71_724d6b515761.slice/crio-90c86e5f76cb7b8264f851fa6864498bde8095fec87c3f643f59557f8bd23d06 WatchSource:0}: Error finding container 90c86e5f76cb7b8264f851fa6864498bde8095fec87c3f643f59557f8bd23d06: Status 404 returned error can't find the container with id 90c86e5f76cb7b8264f851fa6864498bde8095fec87c3f643f59557f8bd23d06
Oct 30 22:11:00.091460 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:00.091437    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-dlh79\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"
Oct 30 22:11:00.091596 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:00.091482    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/df96496e-4bf3-470b-b4d6-8bac68520ab7-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-dlh79\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"
Oct 30 22:11:00.091596 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:00.091522    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/df96496e-4bf3-470b-b4d6-8bac68520ab7-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-dlh79\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"
Oct 30 22:11:00.091596 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:00.091548    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnzz4\" (UniqueName: \"kubernetes.io/projected/df96496e-4bf3-470b-b4d6-8bac68520ab7-kube-api-access-fnzz4\") pod \"istiod-openshift-gateway-7876bfcb74-dlh79\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"
Oct 30 22:11:00.091596 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:00.091586    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-dlh79\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"
Oct 30 22:11:00.091816 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:00.091615    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-dlh79\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"
Oct 30 22:11:00.091816 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:00.091647    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-dlh79\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"
Oct 30 22:11:00.092072 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:00.092048    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-dlh79\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"
Oct 30 22:11:00.094127 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:00.094103    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-dlh79\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"
Oct 30 22:11:00.094266 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:00.094132    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/df96496e-4bf3-470b-b4d6-8bac68520ab7-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-dlh79\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"
Oct 30 22:11:00.094266 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:00.094211    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/df96496e-4bf3-470b-b4d6-8bac68520ab7-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-dlh79\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"
Oct 30 22:11:00.094385 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:00.094320    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-dlh79\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"
Oct 30 22:11:00.112463 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:00.112434    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnzz4\" (UniqueName: \"kubernetes.io/projected/df96496e-4bf3-470b-b4d6-8bac68520ab7-kube-api-access-fnzz4\") pod \"istiod-openshift-gateway-7876bfcb74-dlh79\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"
Oct 30 22:11:00.112716 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:00.112699    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-dlh79\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"
Oct 30 22:11:00.159488 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:00.159452    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:00.216113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:00.216084    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"
Oct 30 22:11:00.531684 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:00.531657    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"]
Oct 30 22:11:00.533212 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:11:00.533180    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf96496e_4bf3_470b_b4d6_8bac68520ab7.slice/crio-87d63b98bc615755a2903d4d053474a0dc452401ac7aed158254de9fe71100cb WatchSource:0}: Error finding container 87d63b98bc615755a2903d4d053474a0dc452401ac7aed158254de9fe71100cb: Status 404 returned error can't find the container with id 87d63b98bc615755a2903d4d053474a0dc452401ac7aed158254de9fe71100cb
Oct 30 22:11:00.787733 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:00.787647    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79" event={"ID":"df96496e-4bf3-470b-b4d6-8bac68520ab7","Type":"ContainerStarted","Data":"87d63b98bc615755a2903d4d053474a0dc452401ac7aed158254de9fe71100cb"}
Oct 30 22:11:00.789105 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:00.789075    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p" event={"ID":"711ee0fc-3e1a-40b5-9a71-724d6b515761","Type":"ContainerStarted","Data":"90c86e5f76cb7b8264f851fa6864498bde8095fec87c3f643f59557f8bd23d06"}
Oct 30 22:11:01.403895 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:01.403858    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:03.135198 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:03.135160    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:11:03.135586 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:03.135241    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:11:03.818577 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:03.818535    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79" event={"ID":"df96496e-4bf3-470b-b4d6-8bac68520ab7","Type":"ContainerStarted","Data":"1073a7603f72e7178d9044eb2360d681250adcfde0b9b02aa98116d2ebb0a6a3"}
Oct 30 22:11:03.818765 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:03.818692    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"
Oct 30 22:11:03.866684 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:03.866621    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79" podStartSLOduration=2.2678220590000002 podStartE2EDuration="4.866604023s" totalImagesPullingTime="2.598781964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:03.864983319 +0000 UTC m=+249.221349643" watchObservedRunningTime="2026-10-30 22:11:03.866604023 +0000 UTC m=+249.222970335"
Oct 30 22:11:04.045969 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:04.045930    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:04.216469 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:04.216383    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:04.825729 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:04.825677    4889 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-dlh79 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:11:04.825916 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:04.825769    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79" podUID="df96496e-4bf3-470b-b4d6-8bac68520ab7" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:11:05.178163 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:05.178032    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:05.357366 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:05.357330    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:05.527145 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:05.527097    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:06.832951 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:06.832921    4889 generic.go:356] "Generic (PLEG): container finished" podID="711ee0fc-3e1a-40b5-9a71-724d6b515761" containerID="061227e897325fbe97890562d38cdb48a8ff0e48393a4dcbc6a8412f3b93d180" exitCode=0
Oct 30 22:11:06.833415 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:06.833001    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p" event={"ID":"711ee0fc-3e1a-40b5-9a71-724d6b515761","Type":"ContainerDied","Data":"061227e897325fbe97890562d38cdb48a8ff0e48393a4dcbc6a8412f3b93d180"}
Oct 30 22:11:07.825342 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:07.825310    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"
Oct 30 22:11:07.838927 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:07.838902    4889 generic.go:356] "Generic (PLEG): container finished" podID="711ee0fc-3e1a-40b5-9a71-724d6b515761" containerID="b1fdf8732ea240fdc304b9277c18b1957ac0ec4b754b849105c55f13ae7df982" exitCode=0
Oct 30 22:11:07.839301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:07.838948    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p" event={"ID":"711ee0fc-3e1a-40b5-9a71-724d6b515761","Type":"ContainerDied","Data":"b1fdf8732ea240fdc304b9277c18b1957ac0ec4b754b849105c55f13ae7df982"}
Oct 30 22:11:07.953396 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:07.953362    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:08.668850 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:08.668811    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:10.304380 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:11:10.304340    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:11:13.868947 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:13.868903    4889 generic.go:356] "Generic (PLEG): container finished" podID="711ee0fc-3e1a-40b5-9a71-724d6b515761" containerID="85c8f59a0029195b4ea97618c29d1f306db949a0f2205d526ba6d2b647ddfc25" exitCode=0
Oct 30 22:11:13.869330 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:13.868964    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p" event={"ID":"711ee0fc-3e1a-40b5-9a71-724d6b515761","Type":"ContainerDied","Data":"85c8f59a0029195b4ea97618c29d1f306db949a0f2205d526ba6d2b647ddfc25"}
Oct 30 22:11:14.999151 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:14.999126    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p"
Oct 30 22:11:15.023395 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:15.023375    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/711ee0fc-3e1a-40b5-9a71-724d6b515761-kube-api-access-l6jhz\" (UniqueName: \"kubernetes.io/projected/711ee0fc-3e1a-40b5-9a71-724d6b515761-kube-api-access-l6jhz\") pod \"711ee0fc-3e1a-40b5-9a71-724d6b515761\" (UID: \"711ee0fc-3e1a-40b5-9a71-724d6b515761\") "
Oct 30 22:11:15.023493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:15.023443    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711ee0fc-3e1a-40b5-9a71-724d6b515761-util\" (UniqueName: \"kubernetes.io/empty-dir/711ee0fc-3e1a-40b5-9a71-724d6b515761-util\") pod \"711ee0fc-3e1a-40b5-9a71-724d6b515761\" (UID: \"711ee0fc-3e1a-40b5-9a71-724d6b515761\") "
Oct 30 22:11:15.023493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:15.023468    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711ee0fc-3e1a-40b5-9a71-724d6b515761-bundle\" (UniqueName: \"kubernetes.io/empty-dir/711ee0fc-3e1a-40b5-9a71-724d6b515761-bundle\") pod \"711ee0fc-3e1a-40b5-9a71-724d6b515761\" (UID: \"711ee0fc-3e1a-40b5-9a71-724d6b515761\") "
Oct 30 22:11:15.023995 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:15.023968    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711ee0fc-3e1a-40b5-9a71-724d6b515761-bundle" pod "711ee0fc-3e1a-40b5-9a71-724d6b515761" (UID: "711ee0fc-3e1a-40b5-9a71-724d6b515761"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:15.025593 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:15.025568    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711ee0fc-3e1a-40b5-9a71-724d6b515761-kube-api-access-l6jhz" pod "711ee0fc-3e1a-40b5-9a71-724d6b515761" (UID: "711ee0fc-3e1a-40b5-9a71-724d6b515761"). InnerVolumeSpecName "kube-api-access-l6jhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:15.028011 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:15.027985    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711ee0fc-3e1a-40b5-9a71-724d6b515761-util" pod "711ee0fc-3e1a-40b5-9a71-724d6b515761" (UID: "711ee0fc-3e1a-40b5-9a71-724d6b515761"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:15.124949 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:15.124875    4889 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/711ee0fc-3e1a-40b5-9a71-724d6b515761-util\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:11:15.124949 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:15.124900    4889 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/711ee0fc-3e1a-40b5-9a71-724d6b515761-bundle\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:11:15.124949 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:15.124909    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6jhz\" (UniqueName: \"kubernetes.io/projected/711ee0fc-3e1a-40b5-9a71-724d6b515761-kube-api-access-l6jhz\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:11:15.879213 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:15.879179    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p" event={"ID":"711ee0fc-3e1a-40b5-9a71-724d6b515761","Type":"ContainerDied","Data":"90c86e5f76cb7b8264f851fa6864498bde8095fec87c3f643f59557f8bd23d06"}
Oct 30 22:11:15.879529 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:15.879241    4889 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90c86e5f76cb7b8264f851fa6864498bde8095fec87c3f643f59557f8bd23d06"
Oct 30 22:11:15.879529 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:15.879242    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jr92p"
Oct 30 22:11:40.300348 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:11:40.300309    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:11:44.856910 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:44.856876    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:53.214270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:53.214235    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:54.781159 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:54.781128    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 22:11:54.790144 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:54.790120    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 22:11:56.388881 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:56.388848    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:56.575334 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:56.575298    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:57.606978 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.606944    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:57.718279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.718249    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2"]
Oct 30 22:11:57.718548 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.718536    4889 cpu_manager.go:414] "containerMap: removing container" podUID="711ee0fc-3e1a-40b5-9a71-724d6b515761" containerName="pull"
Oct 30 22:11:57.718592 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.718550    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711ee0fc-3e1a-40b5-9a71-724d6b515761" containerName="pull"
Oct 30 22:11:57.718592 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.718569    4889 cpu_manager.go:414] "containerMap: removing container" podUID="711ee0fc-3e1a-40b5-9a71-724d6b515761" containerName="extract"
Oct 30 22:11:57.718592 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.718575    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711ee0fc-3e1a-40b5-9a71-724d6b515761" containerName="extract"
Oct 30 22:11:57.718592 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.718588    4889 cpu_manager.go:414] "containerMap: removing container" podUID="711ee0fc-3e1a-40b5-9a71-724d6b515761" containerName="util"
Oct 30 22:11:57.718592 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.718592    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711ee0fc-3e1a-40b5-9a71-724d6b515761" containerName="util"
Oct 30 22:11:57.718748 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.718647    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="711ee0fc-3e1a-40b5-9a71-724d6b515761" containerName="extract"
Oct 30 22:11:57.720493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.720478    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2"
Oct 30 22:11:57.775040 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.775010    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2"]
Oct 30 22:11:57.777437 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.777417    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 22:11:57.777714 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.777691    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 22:11:57.777787 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.777739    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 22:11:57.777829 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.777807    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 22:11:57.777916 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.777901    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 22:11:57.778031 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.778019    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-857vh\""
Oct 30 22:11:57.846654 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.846628    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d09ceb30-c46d-435b-b6e1-b538c6c7822f-manifests\") pod \"opendatahub-operator-controller-manager-c4c4484-h6zs2\" (UID: \"d09ceb30-c46d-435b-b6e1-b538c6c7822f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2"
Oct 30 22:11:57.846822 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.846660    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d09ceb30-c46d-435b-b6e1-b538c6c7822f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c4c4484-h6zs2\" (UID: \"d09ceb30-c46d-435b-b6e1-b538c6c7822f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2"
Oct 30 22:11:57.846822 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.846683    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d09ceb30-c46d-435b-b6e1-b538c6c7822f-webhook-cert\") pod \"opendatahub-operator-controller-manager-c4c4484-h6zs2\" (UID: \"d09ceb30-c46d-435b-b6e1-b538c6c7822f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2"
Oct 30 22:11:57.846822 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.846787    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d09ceb30-c46d-435b-b6e1-b538c6c7822f-metrics-cert\") pod \"opendatahub-operator-controller-manager-c4c4484-h6zs2\" (UID: \"d09ceb30-c46d-435b-b6e1-b538c6c7822f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2"
Oct 30 22:11:57.846986 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.846838    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h47dd\" (UniqueName: \"kubernetes.io/projected/d09ceb30-c46d-435b-b6e1-b538c6c7822f-kube-api-access-h47dd\") pod \"opendatahub-operator-controller-manager-c4c4484-h6zs2\" (UID: \"d09ceb30-c46d-435b-b6e1-b538c6c7822f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2"
Oct 30 22:11:57.847049 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.847018    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d09ceb30-c46d-435b-b6e1-b538c6c7822f-tmp\") pod \"opendatahub-operator-controller-manager-c4c4484-h6zs2\" (UID: \"d09ceb30-c46d-435b-b6e1-b538c6c7822f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2"
Oct 30 22:11:57.948199 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.948122    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h47dd\" (UniqueName: \"kubernetes.io/projected/d09ceb30-c46d-435b-b6e1-b538c6c7822f-kube-api-access-h47dd\") pod \"opendatahub-operator-controller-manager-c4c4484-h6zs2\" (UID: \"d09ceb30-c46d-435b-b6e1-b538c6c7822f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2"
Oct 30 22:11:57.948199 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.948159    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d09ceb30-c46d-435b-b6e1-b538c6c7822f-tmp\") pod \"opendatahub-operator-controller-manager-c4c4484-h6zs2\" (UID: \"d09ceb30-c46d-435b-b6e1-b538c6c7822f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2"
Oct 30 22:11:57.948199 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.948198    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d09ceb30-c46d-435b-b6e1-b538c6c7822f-manifests\") pod \"opendatahub-operator-controller-manager-c4c4484-h6zs2\" (UID: \"d09ceb30-c46d-435b-b6e1-b538c6c7822f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2"
Oct 30 22:11:57.948512 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.948245    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d09ceb30-c46d-435b-b6e1-b538c6c7822f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c4c4484-h6zs2\" (UID: \"d09ceb30-c46d-435b-b6e1-b538c6c7822f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2"
Oct 30 22:11:57.948512 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.948265    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d09ceb30-c46d-435b-b6e1-b538c6c7822f-webhook-cert\") pod \"opendatahub-operator-controller-manager-c4c4484-h6zs2\" (UID: \"d09ceb30-c46d-435b-b6e1-b538c6c7822f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2"
Oct 30 22:11:57.948512 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.948289    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d09ceb30-c46d-435b-b6e1-b538c6c7822f-metrics-cert\") pod \"opendatahub-operator-controller-manager-c4c4484-h6zs2\" (UID: \"d09ceb30-c46d-435b-b6e1-b538c6c7822f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2"
Oct 30 22:11:57.948756 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.948727    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d09ceb30-c46d-435b-b6e1-b538c6c7822f-manifests\") pod \"opendatahub-operator-controller-manager-c4c4484-h6zs2\" (UID: \"d09ceb30-c46d-435b-b6e1-b538c6c7822f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2"
Oct 30 22:11:57.950893 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.950856    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d09ceb30-c46d-435b-b6e1-b538c6c7822f-tmp\") pod \"opendatahub-operator-controller-manager-c4c4484-h6zs2\" (UID: \"d09ceb30-c46d-435b-b6e1-b538c6c7822f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2"
Oct 30 22:11:57.950893 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.950887    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d09ceb30-c46d-435b-b6e1-b538c6c7822f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c4c4484-h6zs2\" (UID: \"d09ceb30-c46d-435b-b6e1-b538c6c7822f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2"
Oct 30 22:11:57.951040 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.950945    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d09ceb30-c46d-435b-b6e1-b538c6c7822f-metrics-cert\") pod \"opendatahub-operator-controller-manager-c4c4484-h6zs2\" (UID: \"d09ceb30-c46d-435b-b6e1-b538c6c7822f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2"
Oct 30 22:11:57.951040 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.951005    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d09ceb30-c46d-435b-b6e1-b538c6c7822f-webhook-cert\") pod \"opendatahub-operator-controller-manager-c4c4484-h6zs2\" (UID: \"d09ceb30-c46d-435b-b6e1-b538c6c7822f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2"
Oct 30 22:11:57.972105 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.972081    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h47dd\" (UniqueName: \"kubernetes.io/projected/d09ceb30-c46d-435b-b6e1-b538c6c7822f-kube-api-access-h47dd\") pod \"opendatahub-operator-controller-manager-c4c4484-h6zs2\" (UID: \"d09ceb30-c46d-435b-b6e1-b538c6c7822f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2"
Oct 30 22:11:57.972658 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:57.972626    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:58.033718 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:58.033694    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2"
Oct 30 22:11:58.148031 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:58.147995    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:58.319911 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:58.319884    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:58.387756 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:58.387721    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2"]
Oct 30 22:11:58.400716 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:11:58.400693    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd09ceb30_c46d_435b_b6e1_b538c6c7822f.slice/crio-6feb3fd9f09f95b536fbe0b2889f9bb6831990e275191556069c83ab93ecd14e WatchSource:0}: Error finding container 6feb3fd9f09f95b536fbe0b2889f9bb6831990e275191556069c83ab93ecd14e: Status 404 returned error can't find the container with id 6feb3fd9f09f95b536fbe0b2889f9bb6831990e275191556069c83ab93ecd14e
Oct 30 22:11:58.402472 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:58.402455    4889 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:11:58.518616 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:58.518576    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:58.709753 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:58.709673    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:59.048054 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:59.047981    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2" event={"ID":"d09ceb30-c46d-435b-b6e1-b538c6c7822f","Type":"ContainerStarted","Data":"6feb3fd9f09f95b536fbe0b2889f9bb6831990e275191556069c83ab93ecd14e"}
Oct 30 22:11:59.099949 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:59.099906    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:59.305417 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:11:59.304617    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:00.224441 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:00.224408    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:00.430951 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:00.430909    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:00.897622 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:00.897586    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:01.066590 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:01.066541    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:02.062823 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:02.062788    4889 generic.go:356] "Generic (PLEG): container finished" podID="d09ceb30-c46d-435b-b6e1-b538c6c7822f" containerID="4b777149d647a4a23732501a7744bb452ff9faf5b86192da4b93d3e19119ee0f" exitCode=0
Oct 30 22:12:02.063246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:02.062879    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2" event={"ID":"d09ceb30-c46d-435b-b6e1-b538c6c7822f","Type":"ContainerDied","Data":"4b777149d647a4a23732501a7744bb452ff9faf5b86192da4b93d3e19119ee0f"}
Oct 30 22:12:02.664409 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:02.664371    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:02.865431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:02.865351    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:03.069204 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:03.069170    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2" event={"ID":"d09ceb30-c46d-435b-b6e1-b538c6c7822f","Type":"ContainerStarted","Data":"c90c59a8ebd589a8496a841fe07d7cd211c27a5d3cbb46e3367b20310eadca99"}
Oct 30 22:12:03.069715 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:03.069270    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2"
Oct 30 22:12:03.155136 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:03.155026    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2" podStartSLOduration=2.560894343 podStartE2EDuration="6.155011976s" totalImagesPullingTime="3.136841097s" totalInitContainerRuntime="457.276536ms" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:03.142421343 +0000 UTC m=+308.498787668" watchObservedRunningTime="2026-10-30 22:12:03.155011976 +0000 UTC m=+308.511378291"
Oct 30 22:12:03.510894 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:03.510863    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:03.714582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:03.714540    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:07.427898 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:07.427820    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:07.874816 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:07.874783    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:09.088318 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:09.088280    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:10.250162 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:10.250129    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:10.299763 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:12:10.299719    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:12:13.925244 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:13.925194    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:14.075492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:14.075462    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-h6zs2"
Oct 30 22:12:14.122637 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:14.122602    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:14.306720 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:14.306685    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:16.683772 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:16.683729    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:17.110288 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:17.110249    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:17.304337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:17.304295    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:18.090075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:18.090043    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:18.538810 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:18.538771    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:18.726670 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:18.726637    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:23.088839 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:23.088800    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:23.265492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:23.265449    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:26.734842 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:26.734809    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:26.942665 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:26.942627    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:27.148547 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:27.148520    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:27.483607 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:27.483525    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:27.687380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:27.687349    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:27.890688 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:27.890655    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:28.203805 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:28.203731    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6f865f994d-9lf4d"]
Oct 30 22:12:28.210084 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:28.210056    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f865f994d-9lf4d"
Oct 30 22:12:28.230143 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:28.230116    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f865f994d-9lf4d"]
Oct 30 22:12:28.257891 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:28.257867    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 22:12:28.258529 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:28.258511    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 22:12:28.258616 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:28.258605    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9nw5f\""
Oct 30 22:12:28.271058 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:28.271025    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:28.292072 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:28.292047    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91d3138c-c623-4574-a608-05088ee9c88b-tmp\") pod \"kube-auth-proxy-6f865f994d-9lf4d\" (UID: \"91d3138c-c623-4574-a608-05088ee9c88b\") " pod="openshift-ingress/kube-auth-proxy-6f865f994d-9lf4d"
Oct 30 22:12:28.292203 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:28.292117    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91d3138c-c623-4574-a608-05088ee9c88b-tls-certs\") pod \"kube-auth-proxy-6f865f994d-9lf4d\" (UID: \"91d3138c-c623-4574-a608-05088ee9c88b\") " pod="openshift-ingress/kube-auth-proxy-6f865f994d-9lf4d"
Oct 30 22:12:28.292297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:28.292249    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msgpf\" (UniqueName: \"kubernetes.io/projected/91d3138c-c623-4574-a608-05088ee9c88b-kube-api-access-msgpf\") pod \"kube-auth-proxy-6f865f994d-9lf4d\" (UID: \"91d3138c-c623-4574-a608-05088ee9c88b\") " pod="openshift-ingress/kube-auth-proxy-6f865f994d-9lf4d"
Oct 30 22:12:28.393012 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:28.392983    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91d3138c-c623-4574-a608-05088ee9c88b-tmp\") pod \"kube-auth-proxy-6f865f994d-9lf4d\" (UID: \"91d3138c-c623-4574-a608-05088ee9c88b\") " pod="openshift-ingress/kube-auth-proxy-6f865f994d-9lf4d"
Oct 30 22:12:28.393179 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:28.393026    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91d3138c-c623-4574-a608-05088ee9c88b-tls-certs\") pod \"kube-auth-proxy-6f865f994d-9lf4d\" (UID: \"91d3138c-c623-4574-a608-05088ee9c88b\") " pod="openshift-ingress/kube-auth-proxy-6f865f994d-9lf4d"
Oct 30 22:12:28.393179 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:28.393072    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msgpf\" (UniqueName: \"kubernetes.io/projected/91d3138c-c623-4574-a608-05088ee9c88b-kube-api-access-msgpf\") pod \"kube-auth-proxy-6f865f994d-9lf4d\" (UID: \"91d3138c-c623-4574-a608-05088ee9c88b\") " pod="openshift-ingress/kube-auth-proxy-6f865f994d-9lf4d"
Oct 30 22:12:28.395357 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:28.395335    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91d3138c-c623-4574-a608-05088ee9c88b-tmp\") pod \"kube-auth-proxy-6f865f994d-9lf4d\" (UID: \"91d3138c-c623-4574-a608-05088ee9c88b\") " pod="openshift-ingress/kube-auth-proxy-6f865f994d-9lf4d"
Oct 30 22:12:28.395861 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:28.395840    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91d3138c-c623-4574-a608-05088ee9c88b-tls-certs\") pod \"kube-auth-proxy-6f865f994d-9lf4d\" (UID: \"91d3138c-c623-4574-a608-05088ee9c88b\") " pod="openshift-ingress/kube-auth-proxy-6f865f994d-9lf4d"
Oct 30 22:12:28.425731 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:28.425696    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msgpf\" (UniqueName: \"kubernetes.io/projected/91d3138c-c623-4574-a608-05088ee9c88b-kube-api-access-msgpf\") pod \"kube-auth-proxy-6f865f994d-9lf4d\" (UID: \"91d3138c-c623-4574-a608-05088ee9c88b\") " pod="openshift-ingress/kube-auth-proxy-6f865f994d-9lf4d"
Oct 30 22:12:28.481706 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:28.481627    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:28.520708 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:28.520671    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f865f994d-9lf4d"
Oct 30 22:12:28.669494 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:28.669462    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:28.825377 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:28.825352    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f865f994d-9lf4d"]
Oct 30 22:12:28.827190 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:12:28.827163    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d3138c_c623_4574_a608_05088ee9c88b.slice/crio-fe516173d9fc2dba3b34a7b674d1028b4eb48b837e1f3bb8a67be7f329fee3c4 WatchSource:0}: Error finding container fe516173d9fc2dba3b34a7b674d1028b4eb48b837e1f3bb8a67be7f329fee3c4: Status 404 returned error can't find the container with id fe516173d9fc2dba3b34a7b674d1028b4eb48b837e1f3bb8a67be7f329fee3c4
Oct 30 22:12:29.170467 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:29.170387    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f865f994d-9lf4d" event={"ID":"91d3138c-c623-4574-a608-05088ee9c88b","Type":"ContainerStarted","Data":"fe516173d9fc2dba3b34a7b674d1028b4eb48b837e1f3bb8a67be7f329fee3c4"}
Oct 30 22:12:29.850716 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:29.850681    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:30.330493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:30.330451    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:30.575309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:30.575277    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:30.768375 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:30.768339    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:31.080690 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:31.080596    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:31.251887 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:31.251857    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:31.433984 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:31.433906    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:32.094278 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:32.094245    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-rsjkj"]
Oct 30 22:12:32.097570 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:32.097553    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rsjkj"
Oct 30 22:12:32.113156 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:32.113139    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 22:12:32.119001 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:32.118969    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdfqc\" (UniqueName: \"kubernetes.io/projected/7d34b350-a707-48de-a9d4-2f073385c366-kube-api-access-hdfqc\") pod \"kserve-module-controller-manager-7566cf5fc8-rsjkj\" (UID: \"7d34b350-a707-48de-a9d4-2f073385c366\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rsjkj"
Oct 30 22:12:32.119112 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:32.119013    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7d34b350-a707-48de-a9d4-2f073385c366-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-rsjkj\" (UID: \"7d34b350-a707-48de-a9d4-2f073385c366\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rsjkj"
Oct 30 22:12:32.119112 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:32.119041    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d34b350-a707-48de-a9d4-2f073385c366-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-rsjkj\" (UID: \"7d34b350-a707-48de-a9d4-2f073385c366\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rsjkj"
Oct 30 22:12:32.119269 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:32.119169    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7d34b350-a707-48de-a9d4-2f073385c366-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-rsjkj\" (UID: \"7d34b350-a707-48de-a9d4-2f073385c366\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rsjkj"
Oct 30 22:12:32.137959 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:32.137927    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-gn8g7\""
Oct 30 22:12:32.150916 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:32.150887    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 22:12:32.151190 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:32.151176    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 22:12:32.152217 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:32.152195    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-rsjkj"]
Oct 30 22:12:32.186761 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:32.186738    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f865f994d-9lf4d" event={"ID":"91d3138c-c623-4574-a608-05088ee9c88b","Type":"ContainerStarted","Data":"71ab601fa8884676d0231ec8374015ee75741e7a997d9260f787b296f19abec2"}
Oct 30 22:12:32.220257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:32.220206    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d34b350-a707-48de-a9d4-2f073385c366-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-rsjkj\" (UID: \"7d34b350-a707-48de-a9d4-2f073385c366\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rsjkj"
Oct 30 22:12:32.220358 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:32.220260    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7d34b350-a707-48de-a9d4-2f073385c366-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-rsjkj\" (UID: \"7d34b350-a707-48de-a9d4-2f073385c366\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rsjkj"
Oct 30 22:12:32.220358 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:32.220321    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdfqc\" (UniqueName: \"kubernetes.io/projected/7d34b350-a707-48de-a9d4-2f073385c366-kube-api-access-hdfqc\") pod \"kserve-module-controller-manager-7566cf5fc8-rsjkj\" (UID: \"7d34b350-a707-48de-a9d4-2f073385c366\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rsjkj"
Oct 30 22:12:32.220457 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:32.220367    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7d34b350-a707-48de-a9d4-2f073385c366-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-rsjkj\" (UID: \"7d34b350-a707-48de-a9d4-2f073385c366\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rsjkj"
Oct 30 22:12:32.220654 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:32.220629    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d34b350-a707-48de-a9d4-2f073385c366-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-rsjkj\" (UID: \"7d34b350-a707-48de-a9d4-2f073385c366\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rsjkj"
Oct 30 22:12:32.220722 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:32.220688    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7d34b350-a707-48de-a9d4-2f073385c366-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-rsjkj\" (UID: \"7d34b350-a707-48de-a9d4-2f073385c366\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rsjkj"
Oct 30 22:12:32.221193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:32.221176    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7d34b350-a707-48de-a9d4-2f073385c366-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-rsjkj\" (UID: \"7d34b350-a707-48de-a9d4-2f073385c366\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rsjkj"
Oct 30 22:12:32.228593 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:32.228547    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6f865f994d-9lf4d" podStartSLOduration=1.9168645789999998 podStartE2EDuration="4.228534977s" totalImagesPullingTime="2.311670398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:32.227391997 +0000 UTC m=+337.583758312" watchObservedRunningTime="2026-10-30 22:12:32.228534977 +0000 UTC m=+337.584901291"
Oct 30 22:12:32.244330 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:32.244300    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdfqc\" (UniqueName: \"kubernetes.io/projected/7d34b350-a707-48de-a9d4-2f073385c366-kube-api-access-hdfqc\") pod \"kserve-module-controller-manager-7566cf5fc8-rsjkj\" (UID: \"7d34b350-a707-48de-a9d4-2f073385c366\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rsjkj"
Oct 30 22:12:32.410453 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:32.410375    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rsjkj"
Oct 30 22:12:32.734075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:32.734043    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-rsjkj"]
Oct 30 22:12:32.735145 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:12:32.735116    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d34b350_a707_48de_a9d4_2f073385c366.slice/crio-e6a5c7f85d750840dfdd1e7b6f723adc47de4f78c9f2e2eb04dd58840fedf758 WatchSource:0}: Error finding container e6a5c7f85d750840dfdd1e7b6f723adc47de4f78c9f2e2eb04dd58840fedf758: Status 404 returned error can't find the container with id e6a5c7f85d750840dfdd1e7b6f723adc47de4f78c9f2e2eb04dd58840fedf758
Oct 30 22:12:33.197060 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:33.197021    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rsjkj" event={"ID":"7d34b350-a707-48de-a9d4-2f073385c366","Type":"ContainerStarted","Data":"e6a5c7f85d750840dfdd1e7b6f723adc47de4f78c9f2e2eb04dd58840fedf758"}
Oct 30 22:12:34.202679 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:34.202636    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rsjkj" event={"ID":"7d34b350-a707-48de-a9d4-2f073385c366","Type":"ContainerStarted","Data":"92126a5045e3047c78f57c0c3f1694306e9ae34d892a2bb9ba8776f62d482607"}
Oct 30 22:12:34.203075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:34.202751    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rsjkj"
Oct 30 22:12:34.256236 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:34.256167    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rsjkj" podStartSLOduration=0.959560892 podStartE2EDuration="2.256153982s" totalImagesPullingTime="1.29659309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:34.255602826 +0000 UTC m=+339.611969153" watchObservedRunningTime="2026-10-30 22:12:34.256153982 +0000 UTC m=+339.612520297"
Oct 30 22:12:35.518747 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:35.518708    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:35.683393 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:35.683358    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:36.507153 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:36.507122    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:36.678044 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:36.678010    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:39.406778 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:39.406741    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:39.576104 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:39.576074    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:40.300359 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:12:40.300308    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:12:41.966065 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:41.966028    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:42.141398 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:42.141362    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:42.348544 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:42.348511    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:42.517102 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:42.517057    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:44.959438 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:44.959405    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:46.697009 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:46.696975    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:46.865993 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:46.865945    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:50.073580 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:50.073548    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:50.265590 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:50.265550    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:50.435243 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:50.435142    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:50.839948 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:50.839910    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:54.805264 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:54.805217    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:54.982761 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:54.982725    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:55.207802 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:12:55.207724    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rsjkj"
Oct 30 22:13:01.381501 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:01.381463    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:01.559367 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:01.559325    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:09.933757 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:09.933726    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-79stc"]
Oct 30 22:13:09.936154 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:09.936135    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-79stc"
Oct 30 22:13:09.971634 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:09.971603    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 22:13:09.971634 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:09.971631    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 22:13:09.971884 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:09.971651    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jthrb\""
Oct 30 22:13:09.971884 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:09.971649    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 22:13:10.007886 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.007855    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpqsj\" (UniqueName: \"kubernetes.io/projected/9bda2dbd-3de0-400c-9f51-e3922c70049a-kube-api-access-gpqsj\") pod \"maas-controller-579cf7c6d-79stc\" (UID: \"9bda2dbd-3de0-400c-9f51-e3922c70049a\") " pod="opendatahub/maas-controller-579cf7c6d-79stc"
Oct 30 22:13:10.008073 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.007917    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9bda2dbd-3de0-400c-9f51-e3922c70049a-webhook-cert\") pod \"maas-controller-579cf7c6d-79stc\" (UID: \"9bda2dbd-3de0-400c-9f51-e3922c70049a\") " pod="opendatahub/maas-controller-579cf7c6d-79stc"
Oct 30 22:13:10.008073 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.008020    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bda2dbd-3de0-400c-9f51-e3922c70049a-metrics-certs\") pod \"maas-controller-579cf7c6d-79stc\" (UID: \"9bda2dbd-3de0-400c-9f51-e3922c70049a\") " pod="opendatahub/maas-controller-579cf7c6d-79stc"
Oct 30 22:13:10.109445 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.109403    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpqsj\" (UniqueName: \"kubernetes.io/projected/9bda2dbd-3de0-400c-9f51-e3922c70049a-kube-api-access-gpqsj\") pod \"maas-controller-579cf7c6d-79stc\" (UID: \"9bda2dbd-3de0-400c-9f51-e3922c70049a\") " pod="opendatahub/maas-controller-579cf7c6d-79stc"
Oct 30 22:13:10.109658 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.109473    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9bda2dbd-3de0-400c-9f51-e3922c70049a-webhook-cert\") pod \"maas-controller-579cf7c6d-79stc\" (UID: \"9bda2dbd-3de0-400c-9f51-e3922c70049a\") " pod="opendatahub/maas-controller-579cf7c6d-79stc"
Oct 30 22:13:10.109658 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.109513    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bda2dbd-3de0-400c-9f51-e3922c70049a-metrics-certs\") pod \"maas-controller-579cf7c6d-79stc\" (UID: \"9bda2dbd-3de0-400c-9f51-e3922c70049a\") " pod="opendatahub/maas-controller-579cf7c6d-79stc"
Oct 30 22:13:10.112183 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.112153    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bda2dbd-3de0-400c-9f51-e3922c70049a-metrics-certs\") pod \"maas-controller-579cf7c6d-79stc\" (UID: \"9bda2dbd-3de0-400c-9f51-e3922c70049a\") " pod="opendatahub/maas-controller-579cf7c6d-79stc"
Oct 30 22:13:10.112322 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.112276    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9bda2dbd-3de0-400c-9f51-e3922c70049a-webhook-cert\") pod \"maas-controller-579cf7c6d-79stc\" (UID: \"9bda2dbd-3de0-400c-9f51-e3922c70049a\") " pod="opendatahub/maas-controller-579cf7c6d-79stc"
Oct 30 22:13:10.131103 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.131077    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpqsj\" (UniqueName: \"kubernetes.io/projected/9bda2dbd-3de0-400c-9f51-e3922c70049a-kube-api-access-gpqsj\") pod \"maas-controller-579cf7c6d-79stc\" (UID: \"9bda2dbd-3de0-400c-9f51-e3922c70049a\") " pod="opendatahub/maas-controller-579cf7c6d-79stc"
Oct 30 22:13:10.247546 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.247504    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-79stc"
Oct 30 22:13:10.300275 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.300246    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8586c445d6-g6ktk"]
Oct 30 22:13:10.302773 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.302751    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8586c445d6-g6ktk"
Oct 30 22:13:10.305242 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:13:10.305193    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:13:10.411817 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.411779    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsc9s\" (UniqueName: \"kubernetes.io/projected/347d81d0-5e87-42c4-89e7-8666cab18419-kube-api-access-fsc9s\") pod \"maas-controller-8586c445d6-g6ktk\" (UID: \"347d81d0-5e87-42c4-89e7-8666cab18419\") " pod="opendatahub/maas-controller-8586c445d6-g6ktk"
Oct 30 22:13:10.411817 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.411821    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/347d81d0-5e87-42c4-89e7-8666cab18419-webhook-cert\") pod \"maas-controller-8586c445d6-g6ktk\" (UID: \"347d81d0-5e87-42c4-89e7-8666cab18419\") " pod="opendatahub/maas-controller-8586c445d6-g6ktk"
Oct 30 22:13:10.412097 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.411871    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/347d81d0-5e87-42c4-89e7-8666cab18419-metrics-certs\") pod \"maas-controller-8586c445d6-g6ktk\" (UID: \"347d81d0-5e87-42c4-89e7-8666cab18419\") " pod="opendatahub/maas-controller-8586c445d6-g6ktk"
Oct 30 22:13:10.513086 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.512994    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/347d81d0-5e87-42c4-89e7-8666cab18419-metrics-certs\") pod \"maas-controller-8586c445d6-g6ktk\" (UID: \"347d81d0-5e87-42c4-89e7-8666cab18419\") " pod="opendatahub/maas-controller-8586c445d6-g6ktk"
Oct 30 22:13:10.513086 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.513051    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsc9s\" (UniqueName: \"kubernetes.io/projected/347d81d0-5e87-42c4-89e7-8666cab18419-kube-api-access-fsc9s\") pod \"maas-controller-8586c445d6-g6ktk\" (UID: \"347d81d0-5e87-42c4-89e7-8666cab18419\") " pod="opendatahub/maas-controller-8586c445d6-g6ktk"
Oct 30 22:13:10.513086 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.513072    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/347d81d0-5e87-42c4-89e7-8666cab18419-webhook-cert\") pod \"maas-controller-8586c445d6-g6ktk\" (UID: \"347d81d0-5e87-42c4-89e7-8666cab18419\") " pod="opendatahub/maas-controller-8586c445d6-g6ktk"
Oct 30 22:13:10.516088 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.516056    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/347d81d0-5e87-42c4-89e7-8666cab18419-metrics-certs\") pod \"maas-controller-8586c445d6-g6ktk\" (UID: \"347d81d0-5e87-42c4-89e7-8666cab18419\") " pod="opendatahub/maas-controller-8586c445d6-g6ktk"
Oct 30 22:13:10.516283 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.516091    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/347d81d0-5e87-42c4-89e7-8666cab18419-webhook-cert\") pod \"maas-controller-8586c445d6-g6ktk\" (UID: \"347d81d0-5e87-42c4-89e7-8666cab18419\") " pod="opendatahub/maas-controller-8586c445d6-g6ktk"
Oct 30 22:13:10.541205 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.541166    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsc9s\" (UniqueName: \"kubernetes.io/projected/347d81d0-5e87-42c4-89e7-8666cab18419-kube-api-access-fsc9s\") pod \"maas-controller-8586c445d6-g6ktk\" (UID: \"347d81d0-5e87-42c4-89e7-8666cab18419\") " pod="opendatahub/maas-controller-8586c445d6-g6ktk"
Oct 30 22:13:10.614112 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.614066    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-79stc"]
Oct 30 22:13:10.615361 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.615337    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8586c445d6-g6ktk"
Oct 30 22:13:10.643744 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.643715    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8586c445d6-g6ktk"]
Oct 30 22:13:10.693783 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.693686    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:10.876528 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.876439    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:10.921470 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.921445    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8586c445d6-g6ktk"]
Oct 30 22:13:10.923281 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:13:10.923247    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod347d81d0_5e87_42c4_89e7_8666cab18419.slice/crio-4a47c1d268a170f167f2dfd1ae76d7d60df2f349a7ba460f0e027701b50718ed WatchSource:0}: Error finding container 4a47c1d268a170f167f2dfd1ae76d7d60df2f349a7ba460f0e027701b50718ed: Status 404 returned error can't find the container with id 4a47c1d268a170f167f2dfd1ae76d7d60df2f349a7ba460f0e027701b50718ed
Oct 30 22:13:10.996199 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:10.996171    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-79stc"]
Oct 30 22:13:10.998063 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:13:10.998037    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bda2dbd_3de0_400c_9f51_e3922c70049a.slice/crio-d3c5c1e849545fa099cc39f08e781a0329f9a78a962dec393aee25b7b7c179cf WatchSource:0}: Error finding container d3c5c1e849545fa099cc39f08e781a0329f9a78a962dec393aee25b7b7c179cf: Status 404 returned error can't find the container with id d3c5c1e849545fa099cc39f08e781a0329f9a78a962dec393aee25b7b7c179cf
Oct 30 22:13:11.356828 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:11.356794    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-79stc" event={"ID":"9bda2dbd-3de0-400c-9f51-e3922c70049a","Type":"ContainerStarted","Data":"d3c5c1e849545fa099cc39f08e781a0329f9a78a962dec393aee25b7b7c179cf"}
Oct 30 22:13:11.357936 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:11.357910    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8586c445d6-g6ktk" event={"ID":"347d81d0-5e87-42c4-89e7-8666cab18419","Type":"ContainerStarted","Data":"4a47c1d268a170f167f2dfd1ae76d7d60df2f349a7ba460f0e027701b50718ed"}
Oct 30 22:13:14.100649 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:14.100569    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:14.282645 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:14.282614    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:14.376380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:14.376284    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-79stc" event={"ID":"9bda2dbd-3de0-400c-9f51-e3922c70049a","Type":"ContainerStarted","Data":"0e1756f8bba00bb6d13d770dd29d564fb5bb9e0d4e8310926559a7feb7f5c356"}
Oct 30 22:13:14.376380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:14.376354    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-79stc"
Oct 30 22:13:14.377702 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:14.377679    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8586c445d6-g6ktk" event={"ID":"347d81d0-5e87-42c4-89e7-8666cab18419","Type":"ContainerStarted","Data":"ee9dd4f65be63b30e3e2e7f58ed50852dc71cf32d25f2955fac0f99265c5d5e9"}
Oct 30 22:13:14.377809 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:14.377739    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8586c445d6-g6ktk"
Oct 30 22:13:14.418567 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:14.418521    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-79stc" podStartSLOduration=2.562391594 podStartE2EDuration="5.418506388s" totalImagesPullingTime="2.856114794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:14.417417239 +0000 UTC m=+379.773783567" watchObservedRunningTime="2026-10-30 22:13:14.418506388 +0000 UTC m=+379.774872703"
Oct 30 22:13:14.460100 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:14.460043    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-8586c445d6-g6ktk" podStartSLOduration=1.615212157 podStartE2EDuration="4.460029012s" totalImagesPullingTime="2.844816855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:14.458768936 +0000 UTC m=+379.815135247" watchObservedRunningTime="2026-10-30 22:13:14.460029012 +0000 UTC m=+379.816395378"
Oct 30 22:13:15.537314 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:15.537265    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:15.712449 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:15.712405    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:25.384923 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:25.384894    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-79stc"
Oct 30 22:13:25.385430 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:25.384963    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8586c445d6-g6ktk"
Oct 30 22:13:25.507556 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:25.507522    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-79stc"]
Oct 30 22:13:25.507765 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:25.507742    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-79stc" podUID="9bda2dbd-3de0-400c-9f51-e3922c70049a" containerName="manager" containerID="cri-o://0e1756f8bba00bb6d13d770dd29d564fb5bb9e0d4e8310926559a7feb7f5c356" gracePeriod=30
Oct 30 22:13:25.755354 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:25.755329    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-79stc"
Oct 30 22:13:25.853697 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:25.853662    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bda2dbd-3de0-400c-9f51-e3922c70049a-webhook-cert\" (UniqueName: \"kubernetes.io/secret/9bda2dbd-3de0-400c-9f51-e3922c70049a-webhook-cert\") pod \"9bda2dbd-3de0-400c-9f51-e3922c70049a\" (UID: \"9bda2dbd-3de0-400c-9f51-e3922c70049a\") "
Oct 30 22:13:25.853887 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:25.853733    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bda2dbd-3de0-400c-9f51-e3922c70049a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bda2dbd-3de0-400c-9f51-e3922c70049a-metrics-certs\") pod \"9bda2dbd-3de0-400c-9f51-e3922c70049a\" (UID: \"9bda2dbd-3de0-400c-9f51-e3922c70049a\") "
Oct 30 22:13:25.853887 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:25.853808    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bda2dbd-3de0-400c-9f51-e3922c70049a-kube-api-access-gpqsj\" (UniqueName: \"kubernetes.io/projected/9bda2dbd-3de0-400c-9f51-e3922c70049a-kube-api-access-gpqsj\") pod \"9bda2dbd-3de0-400c-9f51-e3922c70049a\" (UID: \"9bda2dbd-3de0-400c-9f51-e3922c70049a\") "
Oct 30 22:13:25.856169 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:25.856135    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bda2dbd-3de0-400c-9f51-e3922c70049a-metrics-certs" pod "9bda2dbd-3de0-400c-9f51-e3922c70049a" (UID: "9bda2dbd-3de0-400c-9f51-e3922c70049a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:25.856169 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:25.856149    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bda2dbd-3de0-400c-9f51-e3922c70049a-kube-api-access-gpqsj" pod "9bda2dbd-3de0-400c-9f51-e3922c70049a" (UID: "9bda2dbd-3de0-400c-9f51-e3922c70049a"). InnerVolumeSpecName "kube-api-access-gpqsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:25.856384 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:25.856196    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bda2dbd-3de0-400c-9f51-e3922c70049a-webhook-cert" pod "9bda2dbd-3de0-400c-9f51-e3922c70049a" (UID: "9bda2dbd-3de0-400c-9f51-e3922c70049a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:25.954698 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:25.954618    4889 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9bda2dbd-3de0-400c-9f51-e3922c70049a-webhook-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:13:25.954698 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:25.954647    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bda2dbd-3de0-400c-9f51-e3922c70049a-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:13:25.954698 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:25.954660    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpqsj\" (UniqueName: \"kubernetes.io/projected/9bda2dbd-3de0-400c-9f51-e3922c70049a-kube-api-access-gpqsj\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:13:26.430663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.430627    4889 generic.go:356] "Generic (PLEG): container finished" podID="9bda2dbd-3de0-400c-9f51-e3922c70049a" containerID="0e1756f8bba00bb6d13d770dd29d564fb5bb9e0d4e8310926559a7feb7f5c356" exitCode=0
Oct 30 22:13:26.431107 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.430672    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-79stc"
Oct 30 22:13:26.431107 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.430706    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-79stc" event={"ID":"9bda2dbd-3de0-400c-9f51-e3922c70049a","Type":"ContainerDied","Data":"0e1756f8bba00bb6d13d770dd29d564fb5bb9e0d4e8310926559a7feb7f5c356"}
Oct 30 22:13:26.431107 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.430769    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-79stc" event={"ID":"9bda2dbd-3de0-400c-9f51-e3922c70049a","Type":"ContainerDied","Data":"d3c5c1e849545fa099cc39f08e781a0329f9a78a962dec393aee25b7b7c179cf"}
Oct 30 22:13:26.431107 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.430795    4889 scope.go:122] "RemoveContainer" containerID="0e1756f8bba00bb6d13d770dd29d564fb5bb9e0d4e8310926559a7feb7f5c356"
Oct 30 22:13:26.439809 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.439793    4889 scope.go:122] "RemoveContainer" containerID="0e1756f8bba00bb6d13d770dd29d564fb5bb9e0d4e8310926559a7feb7f5c356"
Oct 30 22:13:26.440074 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:13:26.440058    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e1756f8bba00bb6d13d770dd29d564fb5bb9e0d4e8310926559a7feb7f5c356\": container with ID starting with 0e1756f8bba00bb6d13d770dd29d564fb5bb9e0d4e8310926559a7feb7f5c356 not found: ID does not exist" containerID="0e1756f8bba00bb6d13d770dd29d564fb5bb9e0d4e8310926559a7feb7f5c356"
Oct 30 22:13:26.440116 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.440084    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e1756f8bba00bb6d13d770dd29d564fb5bb9e0d4e8310926559a7feb7f5c356"} err="failed to get container status \"0e1756f8bba00bb6d13d770dd29d564fb5bb9e0d4e8310926559a7feb7f5c356\": rpc error: code = NotFound desc = could not find container \"0e1756f8bba00bb6d13d770dd29d564fb5bb9e0d4e8310926559a7feb7f5c356\": container with ID starting with 0e1756f8bba00bb6d13d770dd29d564fb5bb9e0d4e8310926559a7feb7f5c356 not found: ID does not exist"
Oct 30 22:13:26.497776 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.497744    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-79stc"]
Oct 30 22:13:26.510531 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.510501    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-79stc"]
Oct 30 22:13:26.614765 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.614730    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"]
Oct 30 22:13:26.615119 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.615103    4889 cpu_manager.go:414] "containerMap: removing container" podUID="9bda2dbd-3de0-400c-9f51-e3922c70049a" containerName="manager"
Oct 30 22:13:26.615200 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.615121    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bda2dbd-3de0-400c-9f51-e3922c70049a" containerName="manager"
Oct 30 22:13:26.615328 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.615314    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bda2dbd-3de0-400c-9f51-e3922c70049a" containerName="manager"
Oct 30 22:13:26.619705 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.619684    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"
Oct 30 22:13:26.642009 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.641974    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"]
Oct 30 22:13:26.762674 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.762644    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/07bebf45-5427-4745-bbb8-6ec51bd54042-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c926c\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"
Oct 30 22:13:26.762855 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.762686    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c926c\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"
Oct 30 22:13:26.762855 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.762729    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c926c\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"
Oct 30 22:13:26.762855 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.762830    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c926c\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"
Oct 30 22:13:26.763032 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.762878    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/07bebf45-5427-4745-bbb8-6ec51bd54042-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c926c\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"
Oct 30 22:13:26.763032 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.762975    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c926c\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"
Oct 30 22:13:26.763032 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.763016    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tk25\" (UniqueName: \"kubernetes.io/projected/07bebf45-5427-4745-bbb8-6ec51bd54042-kube-api-access-9tk25\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c926c\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"
Oct 30 22:13:26.772635 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.772611    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bda2dbd-3de0-400c-9f51-e3922c70049a" path="/var/lib/kubelet/pods/9bda2dbd-3de0-400c-9f51-e3922c70049a/volumes"
Oct 30 22:13:26.863918 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.863880    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/07bebf45-5427-4745-bbb8-6ec51bd54042-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c926c\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"
Oct 30 22:13:26.863918 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.863911    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c926c\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"
Oct 30 22:13:26.864197 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.864054    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c926c\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"
Oct 30 22:13:26.864197 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.864112    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c926c\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"
Oct 30 22:13:26.864197 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.864147    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/07bebf45-5427-4745-bbb8-6ec51bd54042-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c926c\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"
Oct 30 22:13:26.864378 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.864208    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c926c\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"
Oct 30 22:13:26.864378 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.864264    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tk25\" (UniqueName: \"kubernetes.io/projected/07bebf45-5427-4745-bbb8-6ec51bd54042-kube-api-access-9tk25\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c926c\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"
Oct 30 22:13:26.865239 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.865203    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c926c\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"
Oct 30 22:13:26.866645 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.866620    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/07bebf45-5427-4745-bbb8-6ec51bd54042-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c926c\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"
Oct 30 22:13:26.866734 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.866647    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c926c\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"
Oct 30 22:13:26.866783 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.866743    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c926c\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"
Oct 30 22:13:26.866783 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.866754    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/07bebf45-5427-4745-bbb8-6ec51bd54042-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c926c\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"
Oct 30 22:13:26.884037 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.884012    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c926c\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"
Oct 30 22:13:26.884410 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.884389    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tk25\" (UniqueName: \"kubernetes.io/projected/07bebf45-5427-4745-bbb8-6ec51bd54042-kube-api-access-9tk25\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c926c\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"
Oct 30 22:13:26.931195 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:26.931161    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"
Oct 30 22:13:27.245154 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:27.245124    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"]
Oct 30 22:13:27.246573 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:13:27.246547    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07bebf45_5427_4745_bbb8_6ec51bd54042.slice/crio-531404b9de387a65c4b63e61e3e01aaf8da23c29ebacf66df39f430d1f0e9b8d WatchSource:0}: Error finding container 531404b9de387a65c4b63e61e3e01aaf8da23c29ebacf66df39f430d1f0e9b8d: Status 404 returned error can't find the container with id 531404b9de387a65c4b63e61e3e01aaf8da23c29ebacf66df39f430d1f0e9b8d
Oct 30 22:13:27.248737 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:27.248706    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:13:27.248816 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:27.248763    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:13:27.438602 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:27.438564    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c" event={"ID":"07bebf45-5427-4745-bbb8-6ec51bd54042","Type":"ContainerStarted","Data":"d3b878eac4db8b9ec3e40abaae4bedf09f667d4270ccdd808a9eb91bab559b3a"}
Oct 30 22:13:27.439121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:27.438623    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c" event={"ID":"07bebf45-5427-4745-bbb8-6ec51bd54042","Type":"ContainerStarted","Data":"531404b9de387a65c4b63e61e3e01aaf8da23c29ebacf66df39f430d1f0e9b8d"}
Oct 30 22:13:27.439121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:27.438649    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"
Oct 30 22:13:27.484188 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:27.484126    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c" podStartSLOduration=1.484109066 podStartE2EDuration="1.484109066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:27.481751002 +0000 UTC m=+392.838117344" watchObservedRunningTime="2026-10-30 22:13:27.484109066 +0000 UTC m=+392.840475383"
Oct 30 22:13:27.492199 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:27.492161    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:27.676973 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:27.676883    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:27.856717 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:27.856681    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:28.025763 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:28.025725    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:28.230402 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:28.230363    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:28.445576 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:28.445492    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"
Oct 30 22:13:28.521157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:28.521121    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"]
Oct 30 22:13:28.521355 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:28.521328    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79" podUID="df96496e-4bf3-470b-b4d6-8bac68520ab7" containerName="discovery" containerID="cri-o://1073a7603f72e7178d9044eb2360d681250adcfde0b9b02aa98116d2ebb0a6a3" gracePeriod=30
Oct 30 22:13:28.784360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:28.784335    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"
Oct 30 22:13:28.885341 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:28.885306    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df96496e-4bf3-470b-b4d6-8bac68520ab7-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/df96496e-4bf3-470b-b4d6-8bac68520ab7-local-certs\") pod \"df96496e-4bf3-470b-b4d6-8bac68520ab7\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") "
Oct 30 22:13:28.885537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:28.885387    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df96496e-4bf3-470b-b4d6-8bac68520ab7-cacerts\" (UniqueName: \"kubernetes.io/secret/df96496e-4bf3-470b-b4d6-8bac68520ab7-cacerts\") pod \"df96496e-4bf3-470b-b4d6-8bac68520ab7\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") "
Oct 30 22:13:28.885537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:28.885450    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-token\" (UniqueName: \"kubernetes.io/projected/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-token\") pod \"df96496e-4bf3-470b-b4d6-8bac68520ab7\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") "
Oct 30 22:13:28.885537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:28.885490    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df96496e-4bf3-470b-b4d6-8bac68520ab7-kube-api-access-fnzz4\" (UniqueName: \"kubernetes.io/projected/df96496e-4bf3-470b-b4d6-8bac68520ab7-kube-api-access-fnzz4\") pod \"df96496e-4bf3-470b-b4d6-8bac68520ab7\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") "
Oct 30 22:13:28.885537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:28.885522    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-kubeconfig\") pod \"df96496e-4bf3-470b-b4d6-8bac68520ab7\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") "
Oct 30 22:13:28.885741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:28.885557    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-csr-ca-configmap\") pod \"df96496e-4bf3-470b-b4d6-8bac68520ab7\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") "
Oct 30 22:13:28.885741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:28.885610    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-csr-dns-cert\") pod \"df96496e-4bf3-470b-b4d6-8bac68520ab7\" (UID: \"df96496e-4bf3-470b-b4d6-8bac68520ab7\") "
Oct 30 22:13:28.885980 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:28.885948    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-csr-ca-configmap" pod "df96496e-4bf3-470b-b4d6-8bac68520ab7" (UID: "df96496e-4bf3-470b-b4d6-8bac68520ab7"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:28.888007 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:28.887979    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df96496e-4bf3-470b-b4d6-8bac68520ab7-cacerts" pod "df96496e-4bf3-470b-b4d6-8bac68520ab7" (UID: "df96496e-4bf3-470b-b4d6-8bac68520ab7"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:28.888117 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:28.888092    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-csr-dns-cert" pod "df96496e-4bf3-470b-b4d6-8bac68520ab7" (UID: "df96496e-4bf3-470b-b4d6-8bac68520ab7"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:28.888182 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:28.888145    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df96496e-4bf3-470b-b4d6-8bac68520ab7-local-certs" pod "df96496e-4bf3-470b-b4d6-8bac68520ab7" (UID: "df96496e-4bf3-470b-b4d6-8bac68520ab7"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:28.889294 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:28.888257    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-token" pod "df96496e-4bf3-470b-b4d6-8bac68520ab7" (UID: "df96496e-4bf3-470b-b4d6-8bac68520ab7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:28.889294 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:28.888348    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df96496e-4bf3-470b-b4d6-8bac68520ab7-kube-api-access-fnzz4" pod "df96496e-4bf3-470b-b4d6-8bac68520ab7" (UID: "df96496e-4bf3-470b-b4d6-8bac68520ab7"). InnerVolumeSpecName "kube-api-access-fnzz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:28.889294 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:28.888505    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-kubeconfig" pod "df96496e-4bf3-470b-b4d6-8bac68520ab7" (UID: "df96496e-4bf3-470b-b4d6-8bac68520ab7"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:28.986867 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:28.986837    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:13:28.986867 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:28.986864    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnzz4\" (UniqueName: \"kubernetes.io/projected/df96496e-4bf3-470b-b4d6-8bac68520ab7-kube-api-access-fnzz4\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:13:28.986867 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:28.986874    4889 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-kubeconfig\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:13:28.987111 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:28.986885    4889 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-csr-ca-configmap\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:13:28.987111 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:28.986893    4889 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/df96496e-4bf3-470b-b4d6-8bac68520ab7-istio-csr-dns-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:13:28.987111 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:28.986903    4889 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/df96496e-4bf3-470b-b4d6-8bac68520ab7-local-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:13:28.987111 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:28.986910    4889 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/df96496e-4bf3-470b-b4d6-8bac68520ab7-cacerts\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:13:29.449788 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:29.449749    4889 generic.go:356] "Generic (PLEG): container finished" podID="df96496e-4bf3-470b-b4d6-8bac68520ab7" containerID="1073a7603f72e7178d9044eb2360d681250adcfde0b9b02aa98116d2ebb0a6a3" exitCode=0
Oct 30 22:13:29.450173 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:29.449799    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"
Oct 30 22:13:29.450173 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:29.449836    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79" event={"ID":"df96496e-4bf3-470b-b4d6-8bac68520ab7","Type":"ContainerDied","Data":"1073a7603f72e7178d9044eb2360d681250adcfde0b9b02aa98116d2ebb0a6a3"}
Oct 30 22:13:29.450173 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:29.449886    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79" event={"ID":"df96496e-4bf3-470b-b4d6-8bac68520ab7","Type":"ContainerDied","Data":"87d63b98bc615755a2903d4d053474a0dc452401ac7aed158254de9fe71100cb"}
Oct 30 22:13:29.450173 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:29.449911    4889 scope.go:122] "RemoveContainer" containerID="1073a7603f72e7178d9044eb2360d681250adcfde0b9b02aa98116d2ebb0a6a3"
Oct 30 22:13:29.459656 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:29.459639    4889 scope.go:122] "RemoveContainer" containerID="1073a7603f72e7178d9044eb2360d681250adcfde0b9b02aa98116d2ebb0a6a3"
Oct 30 22:13:29.459929 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:13:29.459913    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1073a7603f72e7178d9044eb2360d681250adcfde0b9b02aa98116d2ebb0a6a3\": container with ID starting with 1073a7603f72e7178d9044eb2360d681250adcfde0b9b02aa98116d2ebb0a6a3 not found: ID does not exist" containerID="1073a7603f72e7178d9044eb2360d681250adcfde0b9b02aa98116d2ebb0a6a3"
Oct 30 22:13:29.459972 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:29.459940    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1073a7603f72e7178d9044eb2360d681250adcfde0b9b02aa98116d2ebb0a6a3"} err="failed to get container status \"1073a7603f72e7178d9044eb2360d681250adcfde0b9b02aa98116d2ebb0a6a3\": rpc error: code = NotFound desc = could not find container \"1073a7603f72e7178d9044eb2360d681250adcfde0b9b02aa98116d2ebb0a6a3\": container with ID starting with 1073a7603f72e7178d9044eb2360d681250adcfde0b9b02aa98116d2ebb0a6a3 not found: ID does not exist"
Oct 30 22:13:29.518269 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:29.518216    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"]
Oct 30 22:13:29.530769 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:29.530739    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-dlh79"]
Oct 30 22:13:30.083023 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:30.082983    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:30.261038 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:30.261003    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:30.426488 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:30.426405    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:30.598141 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:30.598106    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:30.772061 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:30.772032    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df96496e-4bf3-470b-b4d6-8bac68520ab7" path="/var/lib/kubelet/pods/df96496e-4bf3-470b-b4d6-8bac68520ab7/volumes"
Oct 30 22:13:40.300657 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:13:40.300578    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:13:45.090464 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:45.090428    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:45.940446 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:45.940412    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:46.117277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:46.117219    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:47.325286 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:47.325252    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:47.503947 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:47.503913    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:47.672382 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:47.672285    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:47.801597 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:47.801566    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz"]
Oct 30 22:13:47.801908 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:47.801896    4889 cpu_manager.go:414] "containerMap: removing container" podUID="df96496e-4bf3-470b-b4d6-8bac68520ab7" containerName="discovery"
Oct 30 22:13:47.801954 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:47.801909    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df96496e-4bf3-470b-b4d6-8bac68520ab7" containerName="discovery"
Oct 30 22:13:47.801995 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:47.801988    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="df96496e-4bf3-470b-b4d6-8bac68520ab7" containerName="discovery"
Oct 30 22:13:47.807906 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:47.807887    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz"
Oct 30 22:13:47.823424 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:47.823400    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz"]
Oct 30 22:13:47.834308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:47.834286    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:13:47.834308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:47.834286    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:13:47.837674 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:47.837657    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zl57\" (UniqueName: \"kubernetes.io/projected/49b2f86a-db0e-455f-af20-805b3131ca56-kube-api-access-7zl57\") pod \"maas-api-6d669c7fcd-xd8sz\" (UID: \"49b2f86a-db0e-455f-af20-805b3131ca56\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz"
Oct 30 22:13:47.837784 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:47.837681    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49b2f86a-db0e-455f-af20-805b3131ca56-metrics-certs\") pod \"maas-api-6d669c7fcd-xd8sz\" (UID: \"49b2f86a-db0e-455f-af20-805b3131ca56\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz"
Oct 30 22:13:47.837784 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:47.837697    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b2f86a-db0e-455f-af20-805b3131ca56-openshift-service-ca\") pod \"maas-api-6d669c7fcd-xd8sz\" (UID: \"49b2f86a-db0e-455f-af20-805b3131ca56\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz"
Oct 30 22:13:47.837874 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:47.837795    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49b2f86a-db0e-455f-af20-805b3131ca56-maas-api-tls\") pod \"maas-api-6d669c7fcd-xd8sz\" (UID: \"49b2f86a-db0e-455f-af20-805b3131ca56\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz"
Oct 30 22:13:47.881069 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:47.881036    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l95g7\""
Oct 30 22:13:47.881213 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:47.881171    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 22:13:47.881213 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:47.881179    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 22:13:47.938297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:47.938207    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zl57\" (UniqueName: \"kubernetes.io/projected/49b2f86a-db0e-455f-af20-805b3131ca56-kube-api-access-7zl57\") pod \"maas-api-6d669c7fcd-xd8sz\" (UID: \"49b2f86a-db0e-455f-af20-805b3131ca56\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz"
Oct 30 22:13:47.938297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:47.938252    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49b2f86a-db0e-455f-af20-805b3131ca56-metrics-certs\") pod \"maas-api-6d669c7fcd-xd8sz\" (UID: \"49b2f86a-db0e-455f-af20-805b3131ca56\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz"
Oct 30 22:13:47.938297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:47.938270    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b2f86a-db0e-455f-af20-805b3131ca56-openshift-service-ca\") pod \"maas-api-6d669c7fcd-xd8sz\" (UID: \"49b2f86a-db0e-455f-af20-805b3131ca56\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz"
Oct 30 22:13:47.938557 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:47.938315    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49b2f86a-db0e-455f-af20-805b3131ca56-maas-api-tls\") pod \"maas-api-6d669c7fcd-xd8sz\" (UID: \"49b2f86a-db0e-455f-af20-805b3131ca56\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz"
Oct 30 22:13:47.938938 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:47.938914    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b2f86a-db0e-455f-af20-805b3131ca56-openshift-service-ca\") pod \"maas-api-6d669c7fcd-xd8sz\" (UID: \"49b2f86a-db0e-455f-af20-805b3131ca56\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz"
Oct 30 22:13:47.940867 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:47.940842    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49b2f86a-db0e-455f-af20-805b3131ca56-metrics-certs\") pod \"maas-api-6d669c7fcd-xd8sz\" (UID: \"49b2f86a-db0e-455f-af20-805b3131ca56\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz"
Oct 30 22:13:47.940867 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:47.940854    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49b2f86a-db0e-455f-af20-805b3131ca56-maas-api-tls\") pod \"maas-api-6d669c7fcd-xd8sz\" (UID: \"49b2f86a-db0e-455f-af20-805b3131ca56\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz"
Oct 30 22:13:47.957890 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:47.957869    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zl57\" (UniqueName: \"kubernetes.io/projected/49b2f86a-db0e-455f-af20-805b3131ca56-kube-api-access-7zl57\") pod \"maas-api-6d669c7fcd-xd8sz\" (UID: \"49b2f86a-db0e-455f-af20-805b3131ca56\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz"
Oct 30 22:13:48.120147 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:48.120114    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz"
Oct 30 22:13:48.419816 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:48.419783    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz"]
Oct 30 22:13:48.420600 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:13:48.420578    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b2f86a_db0e_455f_af20_805b3131ca56.slice/crio-e5ef92adbee7cea701e7480b963edea359c18972290726497fb921f2b34ec522 WatchSource:0}: Error finding container e5ef92adbee7cea701e7480b963edea359c18972290726497fb921f2b34ec522: Status 404 returned error can't find the container with id e5ef92adbee7cea701e7480b963edea359c18972290726497fb921f2b34ec522
Oct 30 22:13:48.517153 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:48.517116    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:48.535568 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:48.535535    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz" event={"ID":"49b2f86a-db0e-455f-af20-805b3131ca56","Type":"ContainerStarted","Data":"e5ef92adbee7cea701e7480b963edea359c18972290726497fb921f2b34ec522"}
Oct 30 22:13:48.690980 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:48.690891    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:50.548088 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:50.548053    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz" event={"ID":"49b2f86a-db0e-455f-af20-805b3131ca56","Type":"ContainerStarted","Data":"70834a7acf5099b01c3b56f8fba89559434e6264c2c9d6fb3a21463c0cea246e"}
Oct 30 22:13:50.548591 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:50.548135    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz"
Oct 30 22:13:50.593468 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:50.593415    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz" podStartSLOduration=2.166899184 podStartE2EDuration="3.593400125s" totalImagesPullingTime="1.426500941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:50.592165815 +0000 UTC m=+415.948532129" watchObservedRunningTime="2026-10-30 22:13:50.593400125 +0000 UTC m=+415.949766462"
Oct 30 22:13:56.558028 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:56.558000    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz"
Oct 30 22:13:57.153489 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:57.153456    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:57.572196 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:57.572153    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:57.580946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:57.580918    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj"]
Oct 30 22:13:57.585708 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:57.585687    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj"
Oct 30 22:13:57.614297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:57.614270    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj"]
Oct 30 22:13:57.619211 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:57.619193    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-metrics-certs\") pod \"maas-api-6cf6b4c745-2jdfj\" (UID: \"ca14b6fa-378e-42e3-be2e-3b15de4a82cd\") " pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj"
Oct 30 22:13:57.619338 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:57.619243    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz4fg\" (UniqueName: \"kubernetes.io/projected/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-kube-api-access-gz4fg\") pod \"maas-api-6cf6b4c745-2jdfj\" (UID: \"ca14b6fa-378e-42e3-be2e-3b15de4a82cd\") " pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj"
Oct 30 22:13:57.619401 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:57.619365    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-openshift-service-ca\") pod \"maas-api-6cf6b4c745-2jdfj\" (UID: \"ca14b6fa-378e-42e3-be2e-3b15de4a82cd\") " pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj"
Oct 30 22:13:57.619461 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:57.619428    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-maas-api-tls\") pod \"maas-api-6cf6b4c745-2jdfj\" (UID: \"ca14b6fa-378e-42e3-be2e-3b15de4a82cd\") " pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj"
Oct 30 22:13:57.720724 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:57.720692    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-metrics-certs\") pod \"maas-api-6cf6b4c745-2jdfj\" (UID: \"ca14b6fa-378e-42e3-be2e-3b15de4a82cd\") " pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj"
Oct 30 22:13:57.720896 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:57.720739    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz4fg\" (UniqueName: \"kubernetes.io/projected/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-kube-api-access-gz4fg\") pod \"maas-api-6cf6b4c745-2jdfj\" (UID: \"ca14b6fa-378e-42e3-be2e-3b15de4a82cd\") " pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj"
Oct 30 22:13:57.720896 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:57.720778    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-openshift-service-ca\") pod \"maas-api-6cf6b4c745-2jdfj\" (UID: \"ca14b6fa-378e-42e3-be2e-3b15de4a82cd\") " pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj"
Oct 30 22:13:57.720896 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:57.720815    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-maas-api-tls\") pod \"maas-api-6cf6b4c745-2jdfj\" (UID: \"ca14b6fa-378e-42e3-be2e-3b15de4a82cd\") " pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj"
Oct 30 22:13:57.721589 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:57.721504    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-openshift-service-ca\") pod \"maas-api-6cf6b4c745-2jdfj\" (UID: \"ca14b6fa-378e-42e3-be2e-3b15de4a82cd\") " pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj"
Oct 30 22:13:57.723428 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:57.723405    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-maas-api-tls\") pod \"maas-api-6cf6b4c745-2jdfj\" (UID: \"ca14b6fa-378e-42e3-be2e-3b15de4a82cd\") " pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj"
Oct 30 22:13:57.723520 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:57.723436    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-metrics-certs\") pod \"maas-api-6cf6b4c745-2jdfj\" (UID: \"ca14b6fa-378e-42e3-be2e-3b15de4a82cd\") " pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj"
Oct 30 22:13:57.744268 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:57.744216    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz4fg\" (UniqueName: \"kubernetes.io/projected/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-kube-api-access-gz4fg\") pod \"maas-api-6cf6b4c745-2jdfj\" (UID: \"ca14b6fa-378e-42e3-be2e-3b15de4a82cd\") " pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj"
Oct 30 22:13:57.897956 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:57.897862    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj"
Oct 30 22:13:58.193614 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:58.193584    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj"]
Oct 30 22:13:58.194432 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:13:58.194410    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca14b6fa_378e_42e3_be2e_3b15de4a82cd.slice/crio-de8a814b2b27382ff3cc9b90b088ba8981e7d3ceb05cc21506824251f4d39c5e WatchSource:0}: Error finding container de8a814b2b27382ff3cc9b90b088ba8981e7d3ceb05cc21506824251f4d39c5e: Status 404 returned error can't find the container with id de8a814b2b27382ff3cc9b90b088ba8981e7d3ceb05cc21506824251f4d39c5e
Oct 30 22:13:58.580962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:58.580930    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj" event={"ID":"ca14b6fa-378e-42e3-be2e-3b15de4a82cd","Type":"ContainerStarted","Data":"a09cc8ffb9c7e4adb1b38db395ad6d8348fe61a45e9404d2ba453844e0b84f80"}
Oct 30 22:13:58.581557 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:58.580980    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj" event={"ID":"ca14b6fa-378e-42e3-be2e-3b15de4a82cd","Type":"ContainerStarted","Data":"de8a814b2b27382ff3cc9b90b088ba8981e7d3ceb05cc21506824251f4d39c5e"}
Oct 30 22:13:58.581557 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:58.581011    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj"
Oct 30 22:13:58.622342 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:13:58.622291    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj" podStartSLOduration=1.331048849 podStartE2EDuration="1.622276594s" totalImagesPullingTime="291.227745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:58.621555907 +0000 UTC m=+423.977922222" watchObservedRunningTime="2026-10-30 22:13:58.622276594 +0000 UTC m=+423.978642910"
Oct 30 22:14:04.590466 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:04.590436    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj"
Oct 30 22:14:04.658741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:04.658710    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz"]
Oct 30 22:14:04.658974 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:04.658944    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz" podUID="49b2f86a-db0e-455f-af20-805b3131ca56" containerName="maas-api" containerID="cri-o://70834a7acf5099b01c3b56f8fba89559434e6264c2c9d6fb3a21463c0cea246e" gracePeriod=30
Oct 30 22:14:04.902357 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:04.902329    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz"
Oct 30 22:14:04.980769 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:04.980742    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49b2f86a-db0e-455f-af20-805b3131ca56-metrics-certs\" (UniqueName: \"kubernetes.io/secret/49b2f86a-db0e-455f-af20-805b3131ca56-metrics-certs\") pod \"49b2f86a-db0e-455f-af20-805b3131ca56\" (UID: \"49b2f86a-db0e-455f-af20-805b3131ca56\") "
Oct 30 22:14:04.980925 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:04.980840    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49b2f86a-db0e-455f-af20-805b3131ca56-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b2f86a-db0e-455f-af20-805b3131ca56-openshift-service-ca\") pod \"49b2f86a-db0e-455f-af20-805b3131ca56\" (UID: \"49b2f86a-db0e-455f-af20-805b3131ca56\") "
Oct 30 22:14:04.980925 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:04.980877    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49b2f86a-db0e-455f-af20-805b3131ca56-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49b2f86a-db0e-455f-af20-805b3131ca56-maas-api-tls\") pod \"49b2f86a-db0e-455f-af20-805b3131ca56\" (UID: \"49b2f86a-db0e-455f-af20-805b3131ca56\") "
Oct 30 22:14:04.980925 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:04.980907    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49b2f86a-db0e-455f-af20-805b3131ca56-kube-api-access-7zl57\" (UniqueName: \"kubernetes.io/projected/49b2f86a-db0e-455f-af20-805b3131ca56-kube-api-access-7zl57\") pod \"49b2f86a-db0e-455f-af20-805b3131ca56\" (UID: \"49b2f86a-db0e-455f-af20-805b3131ca56\") "
Oct 30 22:14:04.981237 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:04.981200    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49b2f86a-db0e-455f-af20-805b3131ca56-openshift-service-ca" pod "49b2f86a-db0e-455f-af20-805b3131ca56" (UID: "49b2f86a-db0e-455f-af20-805b3131ca56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:04.983122 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:04.983097    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49b2f86a-db0e-455f-af20-805b3131ca56-metrics-certs" pod "49b2f86a-db0e-455f-af20-805b3131ca56" (UID: "49b2f86a-db0e-455f-af20-805b3131ca56"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:04.983254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:04.983211    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b2f86a-db0e-455f-af20-805b3131ca56-kube-api-access-7zl57" pod "49b2f86a-db0e-455f-af20-805b3131ca56" (UID: "49b2f86a-db0e-455f-af20-805b3131ca56"). InnerVolumeSpecName "kube-api-access-7zl57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:04.983304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:04.983274    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49b2f86a-db0e-455f-af20-805b3131ca56-maas-api-tls" pod "49b2f86a-db0e-455f-af20-805b3131ca56" (UID: "49b2f86a-db0e-455f-af20-805b3131ca56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:05.081449 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:05.081413    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zl57\" (UniqueName: \"kubernetes.io/projected/49b2f86a-db0e-455f-af20-805b3131ca56-kube-api-access-7zl57\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:14:05.081449 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:05.081442    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49b2f86a-db0e-455f-af20-805b3131ca56-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:14:05.081449 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:05.081452    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b2f86a-db0e-455f-af20-805b3131ca56-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:14:05.081449 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:05.081460    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49b2f86a-db0e-455f-af20-805b3131ca56-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:14:05.612562 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:05.612526    4889 generic.go:356] "Generic (PLEG): container finished" podID="49b2f86a-db0e-455f-af20-805b3131ca56" containerID="70834a7acf5099b01c3b56f8fba89559434e6264c2c9d6fb3a21463c0cea246e" exitCode=0
Oct 30 22:14:05.612969 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:05.612594    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz"
Oct 30 22:14:05.612969 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:05.612596    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz" event={"ID":"49b2f86a-db0e-455f-af20-805b3131ca56","Type":"ContainerDied","Data":"70834a7acf5099b01c3b56f8fba89559434e6264c2c9d6fb3a21463c0cea246e"}
Oct 30 22:14:05.612969 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:05.612715    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz" event={"ID":"49b2f86a-db0e-455f-af20-805b3131ca56","Type":"ContainerDied","Data":"e5ef92adbee7cea701e7480b963edea359c18972290726497fb921f2b34ec522"}
Oct 30 22:14:05.612969 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:05.612736    4889 scope.go:122] "RemoveContainer" containerID="70834a7acf5099b01c3b56f8fba89559434e6264c2c9d6fb3a21463c0cea246e"
Oct 30 22:14:05.622962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:05.622905    4889 scope.go:122] "RemoveContainer" containerID="70834a7acf5099b01c3b56f8fba89559434e6264c2c9d6fb3a21463c0cea246e"
Oct 30 22:14:05.623256 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:14:05.623208    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70834a7acf5099b01c3b56f8fba89559434e6264c2c9d6fb3a21463c0cea246e\": container with ID starting with 70834a7acf5099b01c3b56f8fba89559434e6264c2c9d6fb3a21463c0cea246e not found: ID does not exist" containerID="70834a7acf5099b01c3b56f8fba89559434e6264c2c9d6fb3a21463c0cea246e"
Oct 30 22:14:05.623346 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:05.623267    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70834a7acf5099b01c3b56f8fba89559434e6264c2c9d6fb3a21463c0cea246e"} err="failed to get container status \"70834a7acf5099b01c3b56f8fba89559434e6264c2c9d6fb3a21463c0cea246e\": rpc error: code = NotFound desc = could not find container \"70834a7acf5099b01c3b56f8fba89559434e6264c2c9d6fb3a21463c0cea246e\": container with ID starting with 70834a7acf5099b01c3b56f8fba89559434e6264c2c9d6fb3a21463c0cea246e not found: ID does not exist"
Oct 30 22:14:05.663908 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:05.663879    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz"]
Oct 30 22:14:05.667864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:05.667842    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xd8sz"]
Oct 30 22:14:06.772054 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:06.772023    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49b2f86a-db0e-455f-af20-805b3131ca56" path="/var/lib/kubelet/pods/49b2f86a-db0e-455f-af20-805b3131ca56/volumes"
Oct 30 22:14:10.299205 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:14:10.299167    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:14:12.943961 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:12.943906    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:13.111911 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:13.111878    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:40.299971 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:14:40.299931    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:14:45.196115 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:45.196079    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:54.189159 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:54.189120    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:54.376635 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:14:54.376598    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:10.299784 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:15:10.299701    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:15:11.848936 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:11.848887    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97"]
Oct 30 22:15:11.849923 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:11.849899    4889 cpu_manager.go:414] "containerMap: removing container" podUID="49b2f86a-db0e-455f-af20-805b3131ca56" containerName="maas-api"
Oct 30 22:15:11.849923 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:11.849923    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49b2f86a-db0e-455f-af20-805b3131ca56" containerName="maas-api"
Oct 30 22:15:11.850090 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:11.850054    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="49b2f86a-db0e-455f-af20-805b3131ca56" containerName="maas-api"
Oct 30 22:15:11.853087 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:11.853067    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97"
Oct 30 22:15:11.871663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:11.871638    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97"]
Oct 30 22:15:11.882715 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:11.882692    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 22:15:11.882853 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:11.882701    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 22:15:11.882853 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:11.882728    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4qd26\""
Oct 30 22:15:11.882853 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:11.882778    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 22:15:11.949705 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:11.949668    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b491f9ed-944c-48f8-95db-a907d8bfc92c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97\" (UID: \"b491f9ed-944c-48f8-95db-a907d8bfc92c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97"
Oct 30 22:15:11.949705 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:11.949707    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b491f9ed-944c-48f8-95db-a907d8bfc92c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97\" (UID: \"b491f9ed-944c-48f8-95db-a907d8bfc92c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97"
Oct 30 22:15:11.949959 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:11.949813    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b491f9ed-944c-48f8-95db-a907d8bfc92c-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97\" (UID: \"b491f9ed-944c-48f8-95db-a907d8bfc92c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97"
Oct 30 22:15:11.949959 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:11.949883    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b491f9ed-944c-48f8-95db-a907d8bfc92c-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97\" (UID: \"b491f9ed-944c-48f8-95db-a907d8bfc92c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97"
Oct 30 22:15:11.949959 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:11.949948    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b491f9ed-944c-48f8-95db-a907d8bfc92c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97\" (UID: \"b491f9ed-944c-48f8-95db-a907d8bfc92c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97"
Oct 30 22:15:11.950107 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:11.950003    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnxfm\" (UniqueName: \"kubernetes.io/projected/b491f9ed-944c-48f8-95db-a907d8bfc92c-kube-api-access-hnxfm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97\" (UID: \"b491f9ed-944c-48f8-95db-a907d8bfc92c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97"
Oct 30 22:15:11.971476 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:11.971445    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:12.050579 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:12.050545    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b491f9ed-944c-48f8-95db-a907d8bfc92c-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97\" (UID: \"b491f9ed-944c-48f8-95db-a907d8bfc92c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97"
Oct 30 22:15:12.050763 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:12.050586    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b491f9ed-944c-48f8-95db-a907d8bfc92c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97\" (UID: \"b491f9ed-944c-48f8-95db-a907d8bfc92c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97"
Oct 30 22:15:12.050763 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:12.050607    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnxfm\" (UniqueName: \"kubernetes.io/projected/b491f9ed-944c-48f8-95db-a907d8bfc92c-kube-api-access-hnxfm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97\" (UID: \"b491f9ed-944c-48f8-95db-a907d8bfc92c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97"
Oct 30 22:15:12.050763 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:12.050637    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b491f9ed-944c-48f8-95db-a907d8bfc92c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97\" (UID: \"b491f9ed-944c-48f8-95db-a907d8bfc92c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97"
Oct 30 22:15:12.050763 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:12.050661    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b491f9ed-944c-48f8-95db-a907d8bfc92c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97\" (UID: \"b491f9ed-944c-48f8-95db-a907d8bfc92c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97"
Oct 30 22:15:12.050763 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:12.050696    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b491f9ed-944c-48f8-95db-a907d8bfc92c-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97\" (UID: \"b491f9ed-944c-48f8-95db-a907d8bfc92c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97"
Oct 30 22:15:12.051039 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:12.051013    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b491f9ed-944c-48f8-95db-a907d8bfc92c-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97\" (UID: \"b491f9ed-944c-48f8-95db-a907d8bfc92c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97"
Oct 30 22:15:12.051103 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:12.051087    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b491f9ed-944c-48f8-95db-a907d8bfc92c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97\" (UID: \"b491f9ed-944c-48f8-95db-a907d8bfc92c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97"
Oct 30 22:15:12.051160 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:12.051123    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b491f9ed-944c-48f8-95db-a907d8bfc92c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97\" (UID: \"b491f9ed-944c-48f8-95db-a907d8bfc92c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97"
Oct 30 22:15:12.053667 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:12.053646    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b491f9ed-944c-48f8-95db-a907d8bfc92c-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97\" (UID: \"b491f9ed-944c-48f8-95db-a907d8bfc92c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97"
Oct 30 22:15:12.054132 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:12.054109    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b491f9ed-944c-48f8-95db-a907d8bfc92c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97\" (UID: \"b491f9ed-944c-48f8-95db-a907d8bfc92c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97"
Oct 30 22:15:12.072595 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:12.072569    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnxfm\" (UniqueName: \"kubernetes.io/projected/b491f9ed-944c-48f8-95db-a907d8bfc92c-kube-api-access-hnxfm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97\" (UID: \"b491f9ed-944c-48f8-95db-a907d8bfc92c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97"
Oct 30 22:15:12.166899 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:12.166824    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97"
Oct 30 22:15:12.469879 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:12.469851    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97"]
Oct 30 22:15:12.470884 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:15:12.470860    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb491f9ed_944c_48f8_95db_a907d8bfc92c.slice/crio-6b6e13037c946b5d7ab0bb7d62f955ed464f9116bce7ae08e4468fe70f49340d WatchSource:0}: Error finding container 6b6e13037c946b5d7ab0bb7d62f955ed464f9116bce7ae08e4468fe70f49340d: Status 404 returned error can't find the container with id 6b6e13037c946b5d7ab0bb7d62f955ed464f9116bce7ae08e4468fe70f49340d
Oct 30 22:15:12.889784 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:12.889749    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97" event={"ID":"b491f9ed-944c-48f8-95db-a907d8bfc92c","Type":"ContainerStarted","Data":"6b6e13037c946b5d7ab0bb7d62f955ed464f9116bce7ae08e4468fe70f49340d"}
Oct 30 22:15:13.406603 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:13.406566    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"]
Oct 30 22:15:13.410746 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:13.410721    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:15:13.427187 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:13.427159    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"]
Oct 30 22:15:13.437596 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:13.437559    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 22:15:13.464987 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:13.464960    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/437136a8-5481-471f-9ec3-ae970c671a60-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6\" (UID: \"437136a8-5481-471f-9ec3-ae970c671a60\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:15:13.465142 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:13.464996    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/437136a8-5481-471f-9ec3-ae970c671a60-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6\" (UID: \"437136a8-5481-471f-9ec3-ae970c671a60\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:15:13.465142 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:13.465041    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/437136a8-5481-471f-9ec3-ae970c671a60-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6\" (UID: \"437136a8-5481-471f-9ec3-ae970c671a60\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:15:13.465142 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:13.465097    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbxzr\" (UniqueName: \"kubernetes.io/projected/437136a8-5481-471f-9ec3-ae970c671a60-kube-api-access-qbxzr\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6\" (UID: \"437136a8-5481-471f-9ec3-ae970c671a60\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:15:13.465362 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:13.465270    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/437136a8-5481-471f-9ec3-ae970c671a60-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6\" (UID: \"437136a8-5481-471f-9ec3-ae970c671a60\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:15:13.465418 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:13.465367    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/437136a8-5481-471f-9ec3-ae970c671a60-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6\" (UID: \"437136a8-5481-471f-9ec3-ae970c671a60\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:15:13.547958 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:13.547877    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:13.566243 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:13.566197    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/437136a8-5481-471f-9ec3-ae970c671a60-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6\" (UID: \"437136a8-5481-471f-9ec3-ae970c671a60\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:15:13.566414 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:13.566259    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbxzr\" (UniqueName: \"kubernetes.io/projected/437136a8-5481-471f-9ec3-ae970c671a60-kube-api-access-qbxzr\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6\" (UID: \"437136a8-5481-471f-9ec3-ae970c671a60\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:15:13.566414 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:13.566287    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/437136a8-5481-471f-9ec3-ae970c671a60-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6\" (UID: \"437136a8-5481-471f-9ec3-ae970c671a60\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:15:13.566414 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:13.566320    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/437136a8-5481-471f-9ec3-ae970c671a60-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6\" (UID: \"437136a8-5481-471f-9ec3-ae970c671a60\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:15:13.566414 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:13.566349    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/437136a8-5481-471f-9ec3-ae970c671a60-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6\" (UID: \"437136a8-5481-471f-9ec3-ae970c671a60\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:15:13.566414 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:13.566383    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/437136a8-5481-471f-9ec3-ae970c671a60-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6\" (UID: \"437136a8-5481-471f-9ec3-ae970c671a60\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:15:13.566674 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:13.566652    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/437136a8-5481-471f-9ec3-ae970c671a60-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6\" (UID: \"437136a8-5481-471f-9ec3-ae970c671a60\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:15:13.566732 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:13.566670    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/437136a8-5481-471f-9ec3-ae970c671a60-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6\" (UID: \"437136a8-5481-471f-9ec3-ae970c671a60\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:15:13.566732 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:13.566722    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/437136a8-5481-471f-9ec3-ae970c671a60-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6\" (UID: \"437136a8-5481-471f-9ec3-ae970c671a60\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:15:13.569468 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:13.569448    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/437136a8-5481-471f-9ec3-ae970c671a60-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6\" (UID: \"437136a8-5481-471f-9ec3-ae970c671a60\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:15:13.569701 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:13.569671    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/437136a8-5481-471f-9ec3-ae970c671a60-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6\" (UID: \"437136a8-5481-471f-9ec3-ae970c671a60\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:15:13.586968 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:13.586944    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbxzr\" (UniqueName: \"kubernetes.io/projected/437136a8-5481-471f-9ec3-ae970c671a60-kube-api-access-qbxzr\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6\" (UID: \"437136a8-5481-471f-9ec3-ae970c671a60\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:15:13.736268 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:13.736219    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:15:13.897209 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:13.897172    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97" event={"ID":"b491f9ed-944c-48f8-95db-a907d8bfc92c","Type":"ContainerStarted","Data":"fba7fcbfcf7681896edb7cee206b305ed462075264fcd6901e8ec74380266e04"}
Oct 30 22:15:13.939205 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:13.939105    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97" podStartSLOduration=1.7366928480000001 podStartE2EDuration="2.93909003s" totalImagesPullingTime="1.202397182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:13.938995387 +0000 UTC m=+499.295361702" watchObservedRunningTime="2026-10-30 22:15:13.93909003 +0000 UTC m=+499.295456363"
Oct 30 22:15:14.032386 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:14.032358    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"]
Oct 30 22:15:14.032978 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:15:14.032936    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod437136a8_5481_471f_9ec3_ae970c671a60.slice/crio-261cd4a2e8d403583567d49675ac800e2952390dd8eed2306a26baa53f78ee99 WatchSource:0}: Error finding container 261cd4a2e8d403583567d49675ac800e2952390dd8eed2306a26baa53f78ee99: Status 404 returned error can't find the container with id 261cd4a2e8d403583567d49675ac800e2952390dd8eed2306a26baa53f78ee99
Oct 30 22:15:14.167906 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:14.167868    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97"
Oct 30 22:15:14.903188 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:14.903149    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" event={"ID":"437136a8-5481-471f-9ec3-ae970c671a60","Type":"ContainerStarted","Data":"be0e3c6faefb9f4913287dd1a19820c61da5ff9bda3aca14ce68cd4a0270f2ca"}
Oct 30 22:15:14.903579 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:14.903198    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" event={"ID":"437136a8-5481-471f-9ec3-ae970c671a60","Type":"ContainerStarted","Data":"261cd4a2e8d403583567d49675ac800e2952390dd8eed2306a26baa53f78ee99"}
Oct 30 22:15:14.944631 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:14.944575    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podStartSLOduration=1.7719155899999999 podStartE2EDuration="1.944559163s" totalImagesPullingTime="172.643573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:14.942955864 +0000 UTC m=+500.299322182" watchObservedRunningTime="2026-10-30 22:15:14.944559163 +0000 UTC m=+500.300925480"
Oct 30 22:15:15.736622 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:15.736591    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:15:17.115687 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:17.115656    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4"]
Oct 30 22:15:17.119412 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:17.119391    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4"
Oct 30 22:15:17.138031 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:17.138004    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4"]
Oct 30 22:15:17.148416 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:17.148395    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 30 22:15:17.196373 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:17.196348    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef937b28-8694-4a67-8959-8529b98af921-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4\" (UID: \"ef937b28-8694-4a67-8959-8529b98af921\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4"
Oct 30 22:15:17.196497 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:17.196386    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef937b28-8694-4a67-8959-8529b98af921-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4\" (UID: \"ef937b28-8694-4a67-8959-8529b98af921\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4"
Oct 30 22:15:17.196497 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:17.196476    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef937b28-8694-4a67-8959-8529b98af921-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4\" (UID: \"ef937b28-8694-4a67-8959-8529b98af921\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4"
Oct 30 22:15:17.196574 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:17.196515    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef937b28-8694-4a67-8959-8529b98af921-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4\" (UID: \"ef937b28-8694-4a67-8959-8529b98af921\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4"
Oct 30 22:15:17.196574 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:17.196538    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef937b28-8694-4a67-8959-8529b98af921-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4\" (UID: \"ef937b28-8694-4a67-8959-8529b98af921\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4"
Oct 30 22:15:17.196657 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:17.196636    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59f2l\" (UniqueName: \"kubernetes.io/projected/ef937b28-8694-4a67-8959-8529b98af921-kube-api-access-59f2l\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4\" (UID: \"ef937b28-8694-4a67-8959-8529b98af921\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4"
Oct 30 22:15:17.261094 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:17.261052    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:17.297659 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:17.297631    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef937b28-8694-4a67-8959-8529b98af921-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4\" (UID: \"ef937b28-8694-4a67-8959-8529b98af921\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4"
Oct 30 22:15:17.297862 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:17.297667    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef937b28-8694-4a67-8959-8529b98af921-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4\" (UID: \"ef937b28-8694-4a67-8959-8529b98af921\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4"
Oct 30 22:15:17.297862 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:17.297686    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef937b28-8694-4a67-8959-8529b98af921-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4\" (UID: \"ef937b28-8694-4a67-8959-8529b98af921\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4"
Oct 30 22:15:17.297862 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:17.297725    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59f2l\" (UniqueName: \"kubernetes.io/projected/ef937b28-8694-4a67-8959-8529b98af921-kube-api-access-59f2l\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4\" (UID: \"ef937b28-8694-4a67-8959-8529b98af921\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4"
Oct 30 22:15:17.297862 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:17.297766    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef937b28-8694-4a67-8959-8529b98af921-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4\" (UID: \"ef937b28-8694-4a67-8959-8529b98af921\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4"
Oct 30 22:15:17.297862 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:17.297806    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef937b28-8694-4a67-8959-8529b98af921-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4\" (UID: \"ef937b28-8694-4a67-8959-8529b98af921\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4"
Oct 30 22:15:17.298305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:17.298194    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef937b28-8694-4a67-8959-8529b98af921-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4\" (UID: \"ef937b28-8694-4a67-8959-8529b98af921\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4"
Oct 30 22:15:17.298305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:17.298217    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef937b28-8694-4a67-8959-8529b98af921-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4\" (UID: \"ef937b28-8694-4a67-8959-8529b98af921\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4"
Oct 30 22:15:17.298432 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:17.298333    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef937b28-8694-4a67-8959-8529b98af921-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4\" (UID: \"ef937b28-8694-4a67-8959-8529b98af921\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4"
Oct 30 22:15:17.300397 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:17.300373    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef937b28-8694-4a67-8959-8529b98af921-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4\" (UID: \"ef937b28-8694-4a67-8959-8529b98af921\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4"
Oct 30 22:15:17.300799 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:17.300784    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef937b28-8694-4a67-8959-8529b98af921-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4\" (UID: \"ef937b28-8694-4a67-8959-8529b98af921\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4"
Oct 30 22:15:17.318566 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:17.318541    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59f2l\" (UniqueName: \"kubernetes.io/projected/ef937b28-8694-4a67-8959-8529b98af921-kube-api-access-59f2l\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4\" (UID: \"ef937b28-8694-4a67-8959-8529b98af921\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4"
Oct 30 22:15:17.432757 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:17.432683    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4"
Oct 30 22:15:17.730899 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:17.730874    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4"]
Oct 30 22:15:17.731681 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:15:17.731655    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef937b28_8694_4a67_8959_8529b98af921.slice/crio-0e8f5e5c59b070c0963d665d4bac5866fb1b50856c15bc82c4c6d3088cdea6e9 WatchSource:0}: Error finding container 0e8f5e5c59b070c0963d665d4bac5866fb1b50856c15bc82c4c6d3088cdea6e9: Status 404 returned error can't find the container with id 0e8f5e5c59b070c0963d665d4bac5866fb1b50856c15bc82c4c6d3088cdea6e9
Oct 30 22:15:17.917726 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:17.917697    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4" event={"ID":"ef937b28-8694-4a67-8959-8529b98af921","Type":"ContainerStarted","Data":"0e8f5e5c59b070c0963d665d4bac5866fb1b50856c15bc82c4c6d3088cdea6e9"}
Oct 30 22:15:18.923464 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:18.923429    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4" event={"ID":"ef937b28-8694-4a67-8959-8529b98af921","Type":"ContainerStarted","Data":"80d757b3beea49df4c699846a41e7c15ec4c358318949a5fd1b07c00ad611580"}
Oct 30 22:15:18.963129 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:18.963081    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4" podStartSLOduration=1.749358323 podStartE2EDuration="1.963067938s" totalImagesPullingTime="213.709615ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:18.96252548 +0000 UTC m=+504.318891798" watchObservedRunningTime="2026-10-30 22:15:18.963067938 +0000 UTC m=+504.319434253"
Oct 30 22:15:19.252059 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:19.252026    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm"]
Oct 30 22:15:19.255442 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:19.255420    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm"
Oct 30 22:15:19.275598 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:19.275575    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm"]
Oct 30 22:15:19.286031 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:19.286014    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 30 22:15:19.365241 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:19.365187    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:19.418419 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:19.418390    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22e10730-dd08-4c76-8b7c-7b87e23e571d-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pxdsm\" (UID: \"22e10730-dd08-4c76-8b7c-7b87e23e571d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm"
Oct 30 22:15:19.418558 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:19.418430    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r2mm\" (UniqueName: \"kubernetes.io/projected/22e10730-dd08-4c76-8b7c-7b87e23e571d-kube-api-access-4r2mm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pxdsm\" (UID: \"22e10730-dd08-4c76-8b7c-7b87e23e571d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm"
Oct 30 22:15:19.418558 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:19.418530    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22e10730-dd08-4c76-8b7c-7b87e23e571d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pxdsm\" (UID: \"22e10730-dd08-4c76-8b7c-7b87e23e571d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm"
Oct 30 22:15:19.418558 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:19.418552    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22e10730-dd08-4c76-8b7c-7b87e23e571d-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pxdsm\" (UID: \"22e10730-dd08-4c76-8b7c-7b87e23e571d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm"
Oct 30 22:15:19.418684 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:19.418580    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22e10730-dd08-4c76-8b7c-7b87e23e571d-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pxdsm\" (UID: \"22e10730-dd08-4c76-8b7c-7b87e23e571d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm"
Oct 30 22:15:19.418684 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:19.418634    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22e10730-dd08-4c76-8b7c-7b87e23e571d-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pxdsm\" (UID: \"22e10730-dd08-4c76-8b7c-7b87e23e571d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm"
Oct 30 22:15:19.433694 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:19.433670    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4"
Oct 30 22:15:19.520113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:19.520037    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22e10730-dd08-4c76-8b7c-7b87e23e571d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pxdsm\" (UID: \"22e10730-dd08-4c76-8b7c-7b87e23e571d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm"
Oct 30 22:15:19.520113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:19.520076    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22e10730-dd08-4c76-8b7c-7b87e23e571d-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pxdsm\" (UID: \"22e10730-dd08-4c76-8b7c-7b87e23e571d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm"
Oct 30 22:15:19.520113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:19.520099    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22e10730-dd08-4c76-8b7c-7b87e23e571d-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pxdsm\" (UID: \"22e10730-dd08-4c76-8b7c-7b87e23e571d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm"
Oct 30 22:15:19.520353 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:19.520120    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22e10730-dd08-4c76-8b7c-7b87e23e571d-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pxdsm\" (UID: \"22e10730-dd08-4c76-8b7c-7b87e23e571d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm"
Oct 30 22:15:19.520353 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:19.520161    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22e10730-dd08-4c76-8b7c-7b87e23e571d-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pxdsm\" (UID: \"22e10730-dd08-4c76-8b7c-7b87e23e571d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm"
Oct 30 22:15:19.520353 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:19.520200    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r2mm\" (UniqueName: \"kubernetes.io/projected/22e10730-dd08-4c76-8b7c-7b87e23e571d-kube-api-access-4r2mm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pxdsm\" (UID: \"22e10730-dd08-4c76-8b7c-7b87e23e571d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm"
Oct 30 22:15:19.520561 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:19.520538    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22e10730-dd08-4c76-8b7c-7b87e23e571d-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pxdsm\" (UID: \"22e10730-dd08-4c76-8b7c-7b87e23e571d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm"
Oct 30 22:15:19.520625 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:19.520608    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22e10730-dd08-4c76-8b7c-7b87e23e571d-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pxdsm\" (UID: \"22e10730-dd08-4c76-8b7c-7b87e23e571d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm"
Oct 30 22:15:19.520694 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:19.520613    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22e10730-dd08-4c76-8b7c-7b87e23e571d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pxdsm\" (UID: \"22e10730-dd08-4c76-8b7c-7b87e23e571d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm"
Oct 30 22:15:19.522628 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:19.522603    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22e10730-dd08-4c76-8b7c-7b87e23e571d-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pxdsm\" (UID: \"22e10730-dd08-4c76-8b7c-7b87e23e571d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm"
Oct 30 22:15:19.522877 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:19.522860    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22e10730-dd08-4c76-8b7c-7b87e23e571d-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pxdsm\" (UID: \"22e10730-dd08-4c76-8b7c-7b87e23e571d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm"
Oct 30 22:15:19.539834 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:19.539805    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r2mm\" (UniqueName: \"kubernetes.io/projected/22e10730-dd08-4c76-8b7c-7b87e23e571d-kube-api-access-4r2mm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pxdsm\" (UID: \"22e10730-dd08-4c76-8b7c-7b87e23e571d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm"
Oct 30 22:15:19.568252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:19.568209    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm"
Oct 30 22:15:19.864262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:19.864218    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm"]
Oct 30 22:15:19.864996 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:15:19.864968    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e10730_dd08_4c76_8b7c_7b87e23e571d.slice/crio-13dd50f19eb365c7c9970981f3f2212deb4425e67166d8a4fb518ff41e3885fc WatchSource:0}: Error finding container 13dd50f19eb365c7c9970981f3f2212deb4425e67166d8a4fb518ff41e3885fc: Status 404 returned error can't find the container with id 13dd50f19eb365c7c9970981f3f2212deb4425e67166d8a4fb518ff41e3885fc
Oct 30 22:15:19.928801 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:19.928766    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm" event={"ID":"22e10730-dd08-4c76-8b7c-7b87e23e571d","Type":"ContainerStarted","Data":"13dd50f19eb365c7c9970981f3f2212deb4425e67166d8a4fb518ff41e3885fc"}
Oct 30 22:15:20.934564 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:20.934521    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm" event={"ID":"22e10730-dd08-4c76-8b7c-7b87e23e571d","Type":"ContainerStarted","Data":"ec1b1ed1a6a90dccf089b40279c770ac19573524becfad543bcc9c42b49cc1cd"}
Oct 30 22:15:20.975192 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:20.975136    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm" podStartSLOduration=1.795683676 podStartE2EDuration="1.975122837s" totalImagesPullingTime="179.439161ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:20.973764358 +0000 UTC m=+506.330130674" watchObservedRunningTime="2026-10-30 22:15:20.975122837 +0000 UTC m=+506.331489190"
Oct 30 22:15:21.223015 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:21.222934    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:21.568346 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:21.568310    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm"
Oct 30 22:15:22.167740 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:22.167708    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97"
Oct 30 22:15:22.180130 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:22.180106    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97"
Oct 30 22:15:22.424891 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:22.424815    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8"]
Oct 30 22:15:22.428274 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:22.428255    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8"
Oct 30 22:15:22.452870 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:22.452842    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8"]
Oct 30 22:15:22.454481 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:22.454462    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 30 22:15:22.545977 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:22.545943    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba24b6c-74bc-4a07-93c1-ee051a72aba0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qttc8\" (UID: \"eba24b6c-74bc-4a07-93c1-ee051a72aba0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8"
Oct 30 22:15:22.545977 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:22.545979    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eba24b6c-74bc-4a07-93c1-ee051a72aba0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qttc8\" (UID: \"eba24b6c-74bc-4a07-93c1-ee051a72aba0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8"
Oct 30 22:15:22.546173 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:22.545999    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s5bc\" (UniqueName: \"kubernetes.io/projected/eba24b6c-74bc-4a07-93c1-ee051a72aba0-kube-api-access-8s5bc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qttc8\" (UID: \"eba24b6c-74bc-4a07-93c1-ee051a72aba0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8"
Oct 30 22:15:22.546173 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:22.546068    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eba24b6c-74bc-4a07-93c1-ee051a72aba0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qttc8\" (UID: \"eba24b6c-74bc-4a07-93c1-ee051a72aba0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8"
Oct 30 22:15:22.546173 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:22.546129    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eba24b6c-74bc-4a07-93c1-ee051a72aba0-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qttc8\" (UID: \"eba24b6c-74bc-4a07-93c1-ee051a72aba0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8"
Oct 30 22:15:22.546303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:22.546182    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eba24b6c-74bc-4a07-93c1-ee051a72aba0-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qttc8\" (UID: \"eba24b6c-74bc-4a07-93c1-ee051a72aba0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8"
Oct 30 22:15:22.564190 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:22.564156    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:22.647697 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:22.647664    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eba24b6c-74bc-4a07-93c1-ee051a72aba0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qttc8\" (UID: \"eba24b6c-74bc-4a07-93c1-ee051a72aba0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8"
Oct 30 22:15:22.647697 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:22.647699    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s5bc\" (UniqueName: \"kubernetes.io/projected/eba24b6c-74bc-4a07-93c1-ee051a72aba0-kube-api-access-8s5bc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qttc8\" (UID: \"eba24b6c-74bc-4a07-93c1-ee051a72aba0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8"
Oct 30 22:15:22.647928 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:22.647725    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eba24b6c-74bc-4a07-93c1-ee051a72aba0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qttc8\" (UID: \"eba24b6c-74bc-4a07-93c1-ee051a72aba0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8"
Oct 30 22:15:22.647928 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:22.647748    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eba24b6c-74bc-4a07-93c1-ee051a72aba0-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qttc8\" (UID: \"eba24b6c-74bc-4a07-93c1-ee051a72aba0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8"
Oct 30 22:15:22.647928 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:22.647799    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eba24b6c-74bc-4a07-93c1-ee051a72aba0-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qttc8\" (UID: \"eba24b6c-74bc-4a07-93c1-ee051a72aba0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8"
Oct 30 22:15:22.648073 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:22.647959    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba24b6c-74bc-4a07-93c1-ee051a72aba0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qttc8\" (UID: \"eba24b6c-74bc-4a07-93c1-ee051a72aba0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8"
Oct 30 22:15:22.648177 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:22.648155    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eba24b6c-74bc-4a07-93c1-ee051a72aba0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qttc8\" (UID: \"eba24b6c-74bc-4a07-93c1-ee051a72aba0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8"
Oct 30 22:15:22.648217 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:22.648165    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eba24b6c-74bc-4a07-93c1-ee051a72aba0-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qttc8\" (UID: \"eba24b6c-74bc-4a07-93c1-ee051a72aba0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8"
Oct 30 22:15:22.648446 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:22.648425    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eba24b6c-74bc-4a07-93c1-ee051a72aba0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qttc8\" (UID: \"eba24b6c-74bc-4a07-93c1-ee051a72aba0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8"
Oct 30 22:15:22.650245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:22.650210    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eba24b6c-74bc-4a07-93c1-ee051a72aba0-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qttc8\" (UID: \"eba24b6c-74bc-4a07-93c1-ee051a72aba0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8"
Oct 30 22:15:22.650671 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:22.650649    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba24b6c-74bc-4a07-93c1-ee051a72aba0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qttc8\" (UID: \"eba24b6c-74bc-4a07-93c1-ee051a72aba0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8"
Oct 30 22:15:22.667161 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:22.667130    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s5bc\" (UniqueName: \"kubernetes.io/projected/eba24b6c-74bc-4a07-93c1-ee051a72aba0-kube-api-access-8s5bc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qttc8\" (UID: \"eba24b6c-74bc-4a07-93c1-ee051a72aba0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8"
Oct 30 22:15:22.742607 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:22.742580    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8"
Oct 30 22:15:22.954412 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:22.954384    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97"
Oct 30 22:15:23.040359 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:23.040332    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8"]
Oct 30 22:15:23.041534 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:15:23.041505    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba24b6c_74bc_4a07_93c1_ee051a72aba0.slice/crio-e797956394bd662072e699f76e1908fca4ded8e80f8f5cbfdf161107a9f2299f WatchSource:0}: Error finding container e797956394bd662072e699f76e1908fca4ded8e80f8f5cbfdf161107a9f2299f: Status 404 returned error can't find the container with id e797956394bd662072e699f76e1908fca4ded8e80f8f5cbfdf161107a9f2299f
Oct 30 22:15:23.736735 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:23.736702    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:15:23.749318 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:23.749286    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:15:23.949111 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:23.949079    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8" event={"ID":"eba24b6c-74bc-4a07-93c1-ee051a72aba0","Type":"ContainerStarted","Data":"2fbef4fedc542a53cc641394c17ab36a44a798e1e72a1d7c06f7f282e4215577"}
Oct 30 22:15:23.949315 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:23.949129    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8" event={"ID":"eba24b6c-74bc-4a07-93c1-ee051a72aba0","Type":"ContainerStarted","Data":"e797956394bd662072e699f76e1908fca4ded8e80f8f5cbfdf161107a9f2299f"}
Oct 30 22:15:23.960508 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:23.960484    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:15:23.991031 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:23.990994    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8" podStartSLOduration=1.818527355 podStartE2EDuration="1.990979719s" totalImagesPullingTime="172.452364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:23.990099535 +0000 UTC m=+509.346465850" watchObservedRunningTime="2026-10-30 22:15:23.990979719 +0000 UTC m=+509.347346032"
Oct 30 22:15:24.743379 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:24.743346    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8"
Oct 30 22:15:26.515574 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:26.515535    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:27.433515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:27.433480    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4"
Oct 30 22:15:27.446286 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:27.446263    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4"
Oct 30 22:15:27.979321 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:27.979294    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4"
Oct 30 22:15:28.269034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:28.269003    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-8qsr8"]
Oct 30 22:15:28.272822 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:28.272798    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-8qsr8"
Oct 30 22:15:28.293829 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:28.293805    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-8qsr8"]
Oct 30 22:15:28.304415 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:28.304395    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 30 22:15:28.402461 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:28.402429    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2a5fba2-8fc9-4f83-8f0a-996f76b01825-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-8qsr8\" (UID: \"f2a5fba2-8fc9-4f83-8f0a-996f76b01825\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qsr8"
Oct 30 22:15:28.402657 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:28.402485    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2a5fba2-8fc9-4f83-8f0a-996f76b01825-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-8qsr8\" (UID: \"f2a5fba2-8fc9-4f83-8f0a-996f76b01825\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qsr8"
Oct 30 22:15:28.402657 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:28.402586    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbvl6\" (UniqueName: \"kubernetes.io/projected/f2a5fba2-8fc9-4f83-8f0a-996f76b01825-kube-api-access-bbvl6\") pod \"echo-simulator-kserve-64b6f8f96c-8qsr8\" (UID: \"f2a5fba2-8fc9-4f83-8f0a-996f76b01825\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qsr8"
Oct 30 22:15:28.402810 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:28.402682    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2a5fba2-8fc9-4f83-8f0a-996f76b01825-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-8qsr8\" (UID: \"f2a5fba2-8fc9-4f83-8f0a-996f76b01825\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qsr8"
Oct 30 22:15:28.402810 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:28.402736    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2a5fba2-8fc9-4f83-8f0a-996f76b01825-home\") pod \"echo-simulator-kserve-64b6f8f96c-8qsr8\" (UID: \"f2a5fba2-8fc9-4f83-8f0a-996f76b01825\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qsr8"
Oct 30 22:15:28.402901 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:28.402843    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2a5fba2-8fc9-4f83-8f0a-996f76b01825-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-8qsr8\" (UID: \"f2a5fba2-8fc9-4f83-8f0a-996f76b01825\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qsr8"
Oct 30 22:15:28.403294 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:28.403270    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:28.503651 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:28.503535    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2a5fba2-8fc9-4f83-8f0a-996f76b01825-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-8qsr8\" (UID: \"f2a5fba2-8fc9-4f83-8f0a-996f76b01825\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qsr8"
Oct 30 22:15:28.503651 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:28.503578    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2a5fba2-8fc9-4f83-8f0a-996f76b01825-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-8qsr8\" (UID: \"f2a5fba2-8fc9-4f83-8f0a-996f76b01825\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qsr8"
Oct 30 22:15:28.503651 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:28.503603    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbvl6\" (UniqueName: \"kubernetes.io/projected/f2a5fba2-8fc9-4f83-8f0a-996f76b01825-kube-api-access-bbvl6\") pod \"echo-simulator-kserve-64b6f8f96c-8qsr8\" (UID: \"f2a5fba2-8fc9-4f83-8f0a-996f76b01825\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qsr8"
Oct 30 22:15:28.503651 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:28.503657    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2a5fba2-8fc9-4f83-8f0a-996f76b01825-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-8qsr8\" (UID: \"f2a5fba2-8fc9-4f83-8f0a-996f76b01825\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qsr8"
Oct 30 22:15:28.504054 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:28.503715    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2a5fba2-8fc9-4f83-8f0a-996f76b01825-home\") pod \"echo-simulator-kserve-64b6f8f96c-8qsr8\" (UID: \"f2a5fba2-8fc9-4f83-8f0a-996f76b01825\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qsr8"
Oct 30 22:15:28.504054 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:28.503756    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2a5fba2-8fc9-4f83-8f0a-996f76b01825-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-8qsr8\" (UID: \"f2a5fba2-8fc9-4f83-8f0a-996f76b01825\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qsr8"
Oct 30 22:15:28.504167 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:28.504104    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2a5fba2-8fc9-4f83-8f0a-996f76b01825-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-8qsr8\" (UID: \"f2a5fba2-8fc9-4f83-8f0a-996f76b01825\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qsr8"
Oct 30 22:15:28.504309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:28.504282    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2a5fba2-8fc9-4f83-8f0a-996f76b01825-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-8qsr8\" (UID: \"f2a5fba2-8fc9-4f83-8f0a-996f76b01825\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qsr8"
Oct 30 22:15:28.504386 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:28.504293    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2a5fba2-8fc9-4f83-8f0a-996f76b01825-home\") pod \"echo-simulator-kserve-64b6f8f96c-8qsr8\" (UID: \"f2a5fba2-8fc9-4f83-8f0a-996f76b01825\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qsr8"
Oct 30 22:15:28.506475 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:28.506452    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2a5fba2-8fc9-4f83-8f0a-996f76b01825-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-8qsr8\" (UID: \"f2a5fba2-8fc9-4f83-8f0a-996f76b01825\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qsr8"
Oct 30 22:15:28.506760 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:28.506738    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2a5fba2-8fc9-4f83-8f0a-996f76b01825-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-8qsr8\" (UID: \"f2a5fba2-8fc9-4f83-8f0a-996f76b01825\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qsr8"
Oct 30 22:15:28.523195 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:28.523126    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbvl6\" (UniqueName: \"kubernetes.io/projected/f2a5fba2-8fc9-4f83-8f0a-996f76b01825-kube-api-access-bbvl6\") pod \"echo-simulator-kserve-64b6f8f96c-8qsr8\" (UID: \"f2a5fba2-8fc9-4f83-8f0a-996f76b01825\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qsr8"
Oct 30 22:15:28.590404 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:28.590369    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-8qsr8"
Oct 30 22:15:28.886207 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:28.886180    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-8qsr8"]
Oct 30 22:15:28.886703 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:15:28.886678    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a5fba2_8fc9_4f83_8f0a_996f76b01825.slice/crio-11823621341c829e3b6feed6cdef87c2a73f69ad30292c9af0c79cea388df71f WatchSource:0}: Error finding container 11823621341c829e3b6feed6cdef87c2a73f69ad30292c9af0c79cea388df71f: Status 404 returned error can't find the container with id 11823621341c829e3b6feed6cdef87c2a73f69ad30292c9af0c79cea388df71f
Oct 30 22:15:28.973882 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:28.973852    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-8qsr8" event={"ID":"f2a5fba2-8fc9-4f83-8f0a-996f76b01825","Type":"ContainerStarted","Data":"11823621341c829e3b6feed6cdef87c2a73f69ad30292c9af0c79cea388df71f"}
Oct 30 22:15:29.569214 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:29.568512    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm"
Oct 30 22:15:29.584022 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:29.583994    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm"
Oct 30 22:15:29.990597 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:29.990577    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm"
Oct 30 22:15:30.983933 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:30.983894    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-8qsr8" event={"ID":"f2a5fba2-8fc9-4f83-8f0a-996f76b01825","Type":"ContainerStarted","Data":"b3ab032bd990a44ac03b9958bec6a87ec132d78af81e419ebd489f9ce9cefe6a"}
Oct 30 22:15:31.026911 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:31.026850    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-8qsr8" podStartSLOduration=1.561985548 podStartE2EDuration="3.026834573s" totalImagesPullingTime="1.464849025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:31.02611183 +0000 UTC m=+516.382478145" watchObservedRunningTime="2026-10-30 22:15:31.026834573 +0000 UTC m=+516.383200887"
Oct 30 22:15:31.590973 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:31.590939    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-8qsr8"
Oct 30 22:15:32.743274 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:32.743215    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8"
Oct 30 22:15:32.755979 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:32.755954    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8"
Oct 30 22:15:33.004470 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:33.004386    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qttc8"
Oct 30 22:15:38.590708 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:38.590669    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-8qsr8"
Oct 30 22:15:38.603449 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:38.603421    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-8qsr8"
Oct 30 22:15:39.032255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:39.032211    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-8qsr8"
Oct 30 22:15:40.299822 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:15:40.299790    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:15:40.861661 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:40.861626    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-ggxx4"]
Oct 30 22:15:40.865033 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:40.865017    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-ggxx4"
Oct 30 22:15:40.883898 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:40.883871    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-ggxx4"]
Oct 30 22:15:40.890160 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:40.890137    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-vlvcv\""
Oct 30 22:15:40.893911 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:40.893890    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 22:15:40.894089 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:40.894071    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 22:15:40.905090 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:40.905065    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:40.909567 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:40.909544    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h48bg\" (UniqueName: \"kubernetes.io/projected/3598c31c-21dc-420c-9e7e-48e7064360f9-kube-api-access-h48bg\") pod \"ai-gateway-controller-5f6dd4f69f-ggxx4\" (UID: \"3598c31c-21dc-420c-9e7e-48e7064360f9\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-ggxx4"
Oct 30 22:15:40.909650 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:40.909580    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3598c31c-21dc-420c-9e7e-48e7064360f9-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-ggxx4\" (UID: \"3598c31c-21dc-420c-9e7e-48e7064360f9\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-ggxx4"
Oct 30 22:15:41.010347 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:41.010316    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h48bg\" (UniqueName: \"kubernetes.io/projected/3598c31c-21dc-420c-9e7e-48e7064360f9-kube-api-access-h48bg\") pod \"ai-gateway-controller-5f6dd4f69f-ggxx4\" (UID: \"3598c31c-21dc-420c-9e7e-48e7064360f9\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-ggxx4"
Oct 30 22:15:41.010347 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:41.010347    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3598c31c-21dc-420c-9e7e-48e7064360f9-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-ggxx4\" (UID: \"3598c31c-21dc-420c-9e7e-48e7064360f9\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-ggxx4"
Oct 30 22:15:41.012926 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:41.012908    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3598c31c-21dc-420c-9e7e-48e7064360f9-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-ggxx4\" (UID: \"3598c31c-21dc-420c-9e7e-48e7064360f9\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-ggxx4"
Oct 30 22:15:41.029591 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:41.029565    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h48bg\" (UniqueName: \"kubernetes.io/projected/3598c31c-21dc-420c-9e7e-48e7064360f9-kube-api-access-h48bg\") pod \"ai-gateway-controller-5f6dd4f69f-ggxx4\" (UID: \"3598c31c-21dc-420c-9e7e-48e7064360f9\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-ggxx4"
Oct 30 22:15:41.131871 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:41.131784    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:41.178266 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:41.178236    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-ggxx4"
Oct 30 22:15:41.471261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:41.471214    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-ggxx4"]
Oct 30 22:15:41.471924 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:15:41.471895    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3598c31c_21dc_420c_9e7e_48e7064360f9.slice/crio-0dcf4b0fc26e2db1c2814d8fad661aa300dd38a6e924108ed4400812500ca299 WatchSource:0}: Error finding container 0dcf4b0fc26e2db1c2814d8fad661aa300dd38a6e924108ed4400812500ca299: Status 404 returned error can't find the container with id 0dcf4b0fc26e2db1c2814d8fad661aa300dd38a6e924108ed4400812500ca299
Oct 30 22:15:42.036729 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:42.036676    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-ggxx4" event={"ID":"3598c31c-21dc-420c-9e7e-48e7064360f9","Type":"ContainerStarted","Data":"0dcf4b0fc26e2db1c2814d8fad661aa300dd38a6e924108ed4400812500ca299"}
Oct 30 22:15:43.042552 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:43.042521    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-ggxx4" event={"ID":"3598c31c-21dc-420c-9e7e-48e7064360f9","Type":"ContainerStarted","Data":"beddfb33a4111a971db5bb181a3eeebdcec61ce73167947671b01ec208a10875"}
Oct 30 22:15:43.042993 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:43.042642    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-ggxx4"
Oct 30 22:15:43.084607 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:43.084557    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-ggxx4" podStartSLOduration=1.705769751 podStartE2EDuration="3.084545013s" totalImagesPullingTime="1.378775262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:43.083947666 +0000 UTC m=+528.440313978" watchObservedRunningTime="2026-10-30 22:15:43.084545013 +0000 UTC m=+528.440911385"
Oct 30 22:15:44.155833 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:44.155801    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:44.342886 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:44.342846    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:44.361481 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:44.361453    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rp6t6"]
Oct 30 22:15:44.365064 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:44.365039    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-rp6t6"
Oct 30 22:15:44.384874 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:44.384823    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rp6t6"]
Oct 30 22:15:44.395787 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:44.395752    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 22:15:44.395906 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:44.395833    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-psbb2\""
Oct 30 22:15:44.449157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:44.449081    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srldv\" (UniqueName: \"kubernetes.io/projected/f32ffa1e-863c-4df9-b4d1-f80463ed099c-kube-api-access-srldv\") pod \"payload-processing-55fbbbf9dc-rp6t6\" (UID: \"f32ffa1e-863c-4df9-b4d1-f80463ed099c\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rp6t6"
Oct 30 22:15:44.449157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:44.449121    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f32ffa1e-863c-4df9-b4d1-f80463ed099c-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-rp6t6\" (UID: \"f32ffa1e-863c-4df9-b4d1-f80463ed099c\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rp6t6"
Oct 30 22:15:44.550193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:44.550152    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srldv\" (UniqueName: \"kubernetes.io/projected/f32ffa1e-863c-4df9-b4d1-f80463ed099c-kube-api-access-srldv\") pod \"payload-processing-55fbbbf9dc-rp6t6\" (UID: \"f32ffa1e-863c-4df9-b4d1-f80463ed099c\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rp6t6"
Oct 30 22:15:44.550385 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:44.550205    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f32ffa1e-863c-4df9-b4d1-f80463ed099c-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-rp6t6\" (UID: \"f32ffa1e-863c-4df9-b4d1-f80463ed099c\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rp6t6"
Oct 30 22:15:44.550880 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:44.550861    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f32ffa1e-863c-4df9-b4d1-f80463ed099c-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-rp6t6\" (UID: \"f32ffa1e-863c-4df9-b4d1-f80463ed099c\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rp6t6"
Oct 30 22:15:44.571275 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:44.571249    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srldv\" (UniqueName: \"kubernetes.io/projected/f32ffa1e-863c-4df9-b4d1-f80463ed099c-kube-api-access-srldv\") pod \"payload-processing-55fbbbf9dc-rp6t6\" (UID: \"f32ffa1e-863c-4df9-b4d1-f80463ed099c\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rp6t6"
Oct 30 22:15:44.678769 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:44.678734    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-rp6t6"
Oct 30 22:15:44.997024 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:44.996995    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rp6t6"]
Oct 30 22:15:44.997824 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:15:44.997725    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32ffa1e_863c_4df9_b4d1_f80463ed099c.slice/crio-2de57f76d9e4ea1465ab7cd49f0f328094ed80e8d5d95b3943f6ad409569e3db WatchSource:0}: Error finding container 2de57f76d9e4ea1465ab7cd49f0f328094ed80e8d5d95b3943f6ad409569e3db: Status 404 returned error can't find the container with id 2de57f76d9e4ea1465ab7cd49f0f328094ed80e8d5d95b3943f6ad409569e3db
Oct 30 22:15:45.052605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:45.052571    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-rp6t6" event={"ID":"f32ffa1e-863c-4df9-b4d1-f80463ed099c","Type":"ContainerStarted","Data":"2de57f76d9e4ea1465ab7cd49f0f328094ed80e8d5d95b3943f6ad409569e3db"}
Oct 30 22:15:45.350952 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:45.350872    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:47.062811 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:47.062777    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-rp6t6" event={"ID":"f32ffa1e-863c-4df9-b4d1-f80463ed099c","Type":"ContainerStarted","Data":"37bf83729df79a43ae5363648b05c56ce99d795c8962c4ea266664d3a4be602f"}
Oct 30 22:15:47.063250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:47.062899    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-rp6t6"
Oct 30 22:15:47.104306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:47.104268    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-rp6t6" podStartSLOduration=1.127652101 podStartE2EDuration="3.104256949s" totalImagesPullingTime="1.976604848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:47.103330557 +0000 UTC m=+532.459696870" watchObservedRunningTime="2026-10-30 22:15:47.104256949 +0000 UTC m=+532.460623263"
Oct 30 22:15:50.860787 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:50.860750    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"]
Oct 30 22:15:50.864581 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:50.864561    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:50.884030 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:50.884011    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-j98cq\""
Oct 30 22:15:50.897751 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:50.897725    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"]
Oct 30 22:15:50.906599 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:50.906556    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p65ln\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:50.906599 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:50.906591    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p65ln\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:50.906860 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:50.906701    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p65ln\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:50.906860 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:50.906738    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p65ln\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:50.906860 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:50.906816    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5vd9\" (UniqueName: \"kubernetes.io/projected/31604bd8-b5a2-435f-b021-f4f29b31b8ce-kube-api-access-b5vd9\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p65ln\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:50.906860 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:50.906841    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p65ln\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:50.907113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:50.906903    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p65ln\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:50.907113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:50.906929    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p65ln\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:50.907113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:50.906976    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p65ln\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:51.007580 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:51.007554    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5vd9\" (UniqueName: \"kubernetes.io/projected/31604bd8-b5a2-435f-b021-f4f29b31b8ce-kube-api-access-b5vd9\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p65ln\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:51.007580 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:51.007585    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p65ln\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:51.007822 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:51.007637    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p65ln\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:51.007822 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:51.007659    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p65ln\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:51.007822 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:51.007702    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p65ln\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:51.007822 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:51.007736    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p65ln\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:51.007822 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:51.007759    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p65ln\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:51.007822 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:51.007780    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p65ln\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:51.007822 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:51.007815    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p65ln\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:51.008211 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:51.008184    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p65ln\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:51.008316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:51.008244    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p65ln\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:51.008316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:51.008289    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p65ln\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:51.008388 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:51.008329    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p65ln\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:51.008421 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:51.008401    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p65ln\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:51.010330 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:51.010301    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p65ln\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:51.010435 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:51.010352    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p65ln\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:51.027578 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:51.027553    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5vd9\" (UniqueName: \"kubernetes.io/projected/31604bd8-b5a2-435f-b021-f4f29b31b8ce-kube-api-access-b5vd9\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p65ln\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:51.027794 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:51.027777    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p65ln\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:51.178125 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:51.178042    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:51.488129 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:51.488104    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"]
Oct 30 22:15:51.489033 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:15:51.489007    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31604bd8_b5a2_435f_b021_f4f29b31b8ce.slice/crio-1cc63f80ac88f6ce35e880faa84cc51e213c0cd18b173e153ab89f244e205840 WatchSource:0}: Error finding container 1cc63f80ac88f6ce35e880faa84cc51e213c0cd18b173e153ab89f244e205840: Status 404 returned error can't find the container with id 1cc63f80ac88f6ce35e880faa84cc51e213c0cd18b173e153ab89f244e205840
Oct 30 22:15:52.088108 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:52.088069    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln" event={"ID":"31604bd8-b5a2-435f-b021-f4f29b31b8ce","Type":"ContainerStarted","Data":"1cc63f80ac88f6ce35e880faa84cc51e213c0cd18b173e153ab89f244e205840"}
Oct 30 22:15:53.951174 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:53.951143    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:15:53.951549 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:53.951206    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:15:53.951549 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:53.951254    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:15:54.050266 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:54.050218    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-ggxx4"
Oct 30 22:15:54.100631 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:54.100594    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln" event={"ID":"31604bd8-b5a2-435f-b021-f4f29b31b8ce","Type":"ContainerStarted","Data":"bcd65b23813851dffffe16db781dc6ecf5012a5e1e917f91abbe0e7c508121f8"}
Oct 30 22:15:54.143212 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:54.143155    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln" podStartSLOduration=1.6840665910000001 podStartE2EDuration="4.143138474s" totalImagesPullingTime="2.459071883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:54.142054317 +0000 UTC m=+539.498420628" watchObservedRunningTime="2026-10-30 22:15:54.143138474 +0000 UTC m=+539.499504788"
Oct 30 22:15:54.178384 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:54.178353    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:15:55.180109 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:55.180075    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p65ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Oct 30 22:15:55.180518 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:55.180155    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln" podUID="31604bd8-b5a2-435f-b021-f4f29b31b8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Oct 30 22:15:56.178755 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:56.178725    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p65ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Oct 30 22:15:56.178942 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:56.178776    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln" podUID="31604bd8-b5a2-435f-b021-f4f29b31b8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Oct 30 22:15:57.178825 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:57.178795    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p65ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Oct 30 22:15:57.179205 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:57.178842    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln" podUID="31604bd8-b5a2-435f-b021-f4f29b31b8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Oct 30 22:15:58.070431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:58.070400    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-rp6t6"
Oct 30 22:15:58.178939 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:58.178907    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p65ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Oct 30 22:15:58.179286 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:58.178959    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln" podUID="31604bd8-b5a2-435f-b021-f4f29b31b8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Oct 30 22:15:59.178774 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:59.178742    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p65ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Oct 30 22:15:59.178945 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:15:59.178790    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln" podUID="31604bd8-b5a2-435f-b021-f4f29b31b8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Oct 30 22:16:00.179241 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:00.179198    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p65ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Oct 30 22:16:00.179733 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:00.179274    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln" podUID="31604bd8-b5a2-435f-b021-f4f29b31b8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Oct 30 22:16:01.178618 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:01.178587    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p65ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Oct 30 22:16:01.178801 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:01.178631    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln" podUID="31604bd8-b5a2-435f-b021-f4f29b31b8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Oct 30 22:16:02.192417 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:02.192387    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:16:02.193137 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:02.193114    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:16:02.193954 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:02.193934    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:16:04.877543 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:04.877504    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:05.047911 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:05.047880    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:05.901925 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:05.901894    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"]
Oct 30 22:16:05.910000 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:05.909980    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:05.930503 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:05.930481    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"]
Oct 30 22:16:06.050485 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.050456    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cmklv\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.050666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.050503    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246eac1b-e89f-4433-930d-f8adbd60524f-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cmklv\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.050666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.050555    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5ls5\" (UniqueName: \"kubernetes.io/projected/246eac1b-e89f-4433-930d-f8adbd60524f-kube-api-access-z5ls5\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cmklv\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.050666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.050596    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cmklv\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.050666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.050610    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246eac1b-e89f-4433-930d-f8adbd60524f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cmklv\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.050666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.050628    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cmklv\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.050864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.050689    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246eac1b-e89f-4433-930d-f8adbd60524f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cmklv\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.050864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.050775    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cmklv\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.050864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.050806    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cmklv\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.151494 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.151459    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5ls5\" (UniqueName: \"kubernetes.io/projected/246eac1b-e89f-4433-930d-f8adbd60524f-kube-api-access-z5ls5\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cmklv\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.151666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.151519    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cmklv\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.151666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.151537    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246eac1b-e89f-4433-930d-f8adbd60524f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cmklv\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.151666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.151561    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cmklv\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.151666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.151576    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246eac1b-e89f-4433-930d-f8adbd60524f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cmklv\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.151666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.151611    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cmklv\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.151666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.151635    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cmklv\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.151666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.151655    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cmklv\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.152038 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.151718    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246eac1b-e89f-4433-930d-f8adbd60524f-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cmklv\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.152038 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.151963    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cmklv\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.152038 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.152001    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cmklv\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.152200 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.152100    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cmklv\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.152200 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.152133    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cmklv\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.152367 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.152345    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246eac1b-e89f-4433-930d-f8adbd60524f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cmklv\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.154215 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.154194    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cmklv\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.154317 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.154257    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246eac1b-e89f-4433-930d-f8adbd60524f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cmklv\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.172045 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.172012    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246eac1b-e89f-4433-930d-f8adbd60524f-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cmklv\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.172260 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.172242    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5ls5\" (UniqueName: \"kubernetes.io/projected/246eac1b-e89f-4433-930d-f8adbd60524f-kube-api-access-z5ls5\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cmklv\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.222657 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.222625    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:06.533111 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.533083    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"]
Oct 30 22:16:06.533589 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:16:06.533566    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246eac1b_e89f_4433_930d_f8adbd60524f.slice/crio-94522c7c6942013fce77c769aaa987890051fb5f4c6c0c7045776068c194380f WatchSource:0}: Error finding container 94522c7c6942013fce77c769aaa987890051fb5f4c6c0c7045776068c194380f: Status 404 returned error can't find the container with id 94522c7c6942013fce77c769aaa987890051fb5f4c6c0c7045776068c194380f
Oct 30 22:16:06.535699 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.535670    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:16:06.535789 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.535730    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:16:06.535789 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:06.535756    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:16:07.186011 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:07.185964    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv" event={"ID":"246eac1b-e89f-4433-930d-f8adbd60524f","Type":"ContainerStarted","Data":"e860586c0e376bb1fd31759e996d438d196fe3c6e27296b100e7ce4ff813beac"}
Oct 30 22:16:07.186413 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:07.186019    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv" event={"ID":"246eac1b-e89f-4433-930d-f8adbd60524f","Type":"ContainerStarted","Data":"94522c7c6942013fce77c769aaa987890051fb5f4c6c0c7045776068c194380f"}
Oct 30 22:16:07.232209 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:07.232131    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:07.234390 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:07.234353    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cmklv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 30 22:16:07.234544 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:07.234404    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv" podUID="246eac1b-e89f-4433-930d-f8adbd60524f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 30 22:16:08.234250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:08.231622    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cmklv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 30 22:16:08.234250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:08.231691    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv" podUID="246eac1b-e89f-4433-930d-f8adbd60524f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 30 22:16:09.224267 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:09.223575    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cmklv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 30 22:16:09.224267 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:09.223638    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv" podUID="246eac1b-e89f-4433-930d-f8adbd60524f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 30 22:16:09.339576 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:09.339496    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv" podStartSLOduration=4.3394733500000005 podStartE2EDuration="4.33947335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:07.263392291 +0000 UTC m=+552.619758602" watchObservedRunningTime="2026-10-30 22:16:09.33947335 +0000 UTC m=+554.695839665"
Oct 30 22:16:09.340811 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:09.340787    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-ggxx4"]
Oct 30 22:16:09.341203 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:09.341177    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-ggxx4" podUID="3598c31c-21dc-420c-9e7e-48e7064360f9" containerName="manager" containerID="cri-o://beddfb33a4111a971db5bb181a3eeebdcec61ce73167947671b01ec208a10875" gracePeriod=30
Oct 30 22:16:09.605755 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:09.605735    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-ggxx4"
Oct 30 22:16:09.684737 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:09.684688    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3598c31c-21dc-420c-9e7e-48e7064360f9-kube-api-access-h48bg\" (UniqueName: \"kubernetes.io/projected/3598c31c-21dc-420c-9e7e-48e7064360f9-kube-api-access-h48bg\") pod \"3598c31c-21dc-420c-9e7e-48e7064360f9\" (UID: \"3598c31c-21dc-420c-9e7e-48e7064360f9\") "
Oct 30 22:16:09.684737 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:09.684742    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3598c31c-21dc-420c-9e7e-48e7064360f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3598c31c-21dc-420c-9e7e-48e7064360f9-metrics-certs\") pod \"3598c31c-21dc-420c-9e7e-48e7064360f9\" (UID: \"3598c31c-21dc-420c-9e7e-48e7064360f9\") "
Oct 30 22:16:09.687009 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:09.686979    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3598c31c-21dc-420c-9e7e-48e7064360f9-metrics-certs" pod "3598c31c-21dc-420c-9e7e-48e7064360f9" (UID: "3598c31c-21dc-420c-9e7e-48e7064360f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:16:09.687121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:09.687014    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3598c31c-21dc-420c-9e7e-48e7064360f9-kube-api-access-h48bg" pod "3598c31c-21dc-420c-9e7e-48e7064360f9" (UID: "3598c31c-21dc-420c-9e7e-48e7064360f9"). InnerVolumeSpecName "kube-api-access-h48bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:09.786126 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:09.786046    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h48bg\" (UniqueName: \"kubernetes.io/projected/3598c31c-21dc-420c-9e7e-48e7064360f9-kube-api-access-h48bg\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:16:09.786126 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:09.786075    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3598c31c-21dc-420c-9e7e-48e7064360f9-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:16:10.201426 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:10.201347    4889 generic.go:356] "Generic (PLEG): container finished" podID="3598c31c-21dc-420c-9e7e-48e7064360f9" containerID="beddfb33a4111a971db5bb181a3eeebdcec61ce73167947671b01ec208a10875" exitCode=0
Oct 30 22:16:10.201426 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:10.201385    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-ggxx4" event={"ID":"3598c31c-21dc-420c-9e7e-48e7064360f9","Type":"ContainerDied","Data":"beddfb33a4111a971db5bb181a3eeebdcec61ce73167947671b01ec208a10875"}
Oct 30 22:16:10.201426 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:10.201390    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-ggxx4"
Oct 30 22:16:10.201426 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:10.201424    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-ggxx4" event={"ID":"3598c31c-21dc-420c-9e7e-48e7064360f9","Type":"ContainerDied","Data":"0dcf4b0fc26e2db1c2814d8fad661aa300dd38a6e924108ed4400812500ca299"}
Oct 30 22:16:10.201703 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:10.201445    4889 scope.go:122] "RemoveContainer" containerID="beddfb33a4111a971db5bb181a3eeebdcec61ce73167947671b01ec208a10875"
Oct 30 22:16:10.219589 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:10.219568    4889 scope.go:122] "RemoveContainer" containerID="beddfb33a4111a971db5bb181a3eeebdcec61ce73167947671b01ec208a10875"
Oct 30 22:16:10.223885 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:10.223521    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cmklv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 30 22:16:10.223885 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:10.223568    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv" podUID="246eac1b-e89f-4433-930d-f8adbd60524f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 30 22:16:10.228413 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:16:10.228339    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beddfb33a4111a971db5bb181a3eeebdcec61ce73167947671b01ec208a10875\": container with ID starting with beddfb33a4111a971db5bb181a3eeebdcec61ce73167947671b01ec208a10875 not found: ID does not exist" containerID="beddfb33a4111a971db5bb181a3eeebdcec61ce73167947671b01ec208a10875"
Oct 30 22:16:10.228413 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:10.228376    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beddfb33a4111a971db5bb181a3eeebdcec61ce73167947671b01ec208a10875"} err="failed to get container status \"beddfb33a4111a971db5bb181a3eeebdcec61ce73167947671b01ec208a10875\": rpc error: code = NotFound desc = could not find container \"beddfb33a4111a971db5bb181a3eeebdcec61ce73167947671b01ec208a10875\": container with ID starting with beddfb33a4111a971db5bb181a3eeebdcec61ce73167947671b01ec208a10875 not found: ID does not exist"
Oct 30 22:16:10.258174 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:10.258147    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-ggxx4"]
Oct 30 22:16:10.263163 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:10.263139    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-ggxx4"]
Oct 30 22:16:10.299684 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:16:10.299651    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:16:10.772474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:10.772445    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3598c31c-21dc-420c-9e7e-48e7064360f9" path="/var/lib/kubelet/pods/3598c31c-21dc-420c-9e7e-48e7064360f9/volumes"
Oct 30 22:16:11.040572 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:11.040472    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:11.228600 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:11.228567    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cmklv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 30 22:16:11.228879 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:11.228848    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv" podUID="246eac1b-e89f-4433-930d-f8adbd60524f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 30 22:16:11.566780 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:11.566740    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:12.233256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:12.232650    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cmklv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 30 22:16:12.233256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:12.232764    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv" podUID="246eac1b-e89f-4433-930d-f8adbd60524f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 30 22:16:13.147451 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:13.147416    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:13.222947 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:13.222916    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cmklv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 30 22:16:13.223141 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:13.223001    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv" podUID="246eac1b-e89f-4433-930d-f8adbd60524f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 30 22:16:13.324034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:13.324003    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:13.498143 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:13.498085    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:14.697338 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:14.697303    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:14.873117 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:14.873081    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:14.955502 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:14.955374    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:14.955685 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:14.955670    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:14.956383 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:14.956362    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:16:18.228466 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:18.228436    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:28.793262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:28.793214    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:36.292219 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.292138    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:36.459796 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.459758    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:36.804167 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.804134    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"]
Oct 30 22:16:36.804514 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.804500    4889 cpu_manager.go:414] "containerMap: removing container" podUID="3598c31c-21dc-420c-9e7e-48e7064360f9" containerName="manager"
Oct 30 22:16:36.804582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.804514    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3598c31c-21dc-420c-9e7e-48e7064360f9" containerName="manager"
Oct 30 22:16:36.804635 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.804596    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="3598c31c-21dc-420c-9e7e-48e7064360f9" containerName="manager"
Oct 30 22:16:36.815595 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.815572    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:36.828984 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.828953    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"]
Oct 30 22:16:36.836771 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.836747    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:36.836869 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.836785    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32e978e1-1585-4947-8975-fe20d4942bc5-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:36.836869 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.836806    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:36.836939 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.836910    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:36.837014 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.836999    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32e978e1-1585-4947-8975-fe20d4942bc5-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:36.837055 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.837036    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:36.837102 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.837078    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6lhn\" (UniqueName: \"kubernetes.io/projected/32e978e1-1585-4947-8975-fe20d4942bc5-kube-api-access-w6lhn\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:36.837195 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.837177    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:36.837290 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.837216    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32e978e1-1585-4947-8975-fe20d4942bc5-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:36.839115 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.839096    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-d52c59-openshift-default-dockercfg-x7nqp\""
Oct 30 22:16:36.903356 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.903310    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:36.938024 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.937991    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32e978e1-1585-4947-8975-fe20d4942bc5-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:36.938200 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.938043    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:36.938200 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.938069    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6lhn\" (UniqueName: \"kubernetes.io/projected/32e978e1-1585-4947-8975-fe20d4942bc5-kube-api-access-w6lhn\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:36.938200 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.938103    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:36.938200 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.938134    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32e978e1-1585-4947-8975-fe20d4942bc5-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:36.938200 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.938182    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:36.938488 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.938252    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32e978e1-1585-4947-8975-fe20d4942bc5-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:36.938488 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.938287    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:36.938488 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.938336    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:36.938634 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.938579    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:36.939079 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.938852    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:36.939079 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.938892    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32e978e1-1585-4947-8975-fe20d4942bc5-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:36.939079 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.939034    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:36.939079 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.939071    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:36.941169 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.941143    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:36.941296 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.941202    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32e978e1-1585-4947-8975-fe20d4942bc5-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:36.962666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.962638    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32e978e1-1585-4947-8975-fe20d4942bc5-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:36.962804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:36.962767    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6lhn\" (UniqueName: \"kubernetes.io/projected/32e978e1-1585-4947-8975-fe20d4942bc5-kube-api-access-w6lhn\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:37.060110 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.060034    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"]
Oct 30 22:16:37.064046 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.064022    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.087036 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.087010    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:37.088007 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.087980    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"]
Oct 30 22:16:37.097520 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.097502    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0ad7c5-openshift-default-dockercfg-b9j97\""
Oct 30 22:16:37.139637 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.139616    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:37.140133 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.140115    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7221740b-e003-4af1-86a2-396e32fff7ca-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.140213 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.140148    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.140213 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.140181    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7221740b-e003-4af1-86a2-396e32fff7ca-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.140342 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.140321    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7221740b-e003-4af1-86a2-396e32fff7ca-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.140396 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.140350    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.140396 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.140375    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.140499 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.140485    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgbhh\" (UniqueName: \"kubernetes.io/projected/7221740b-e003-4af1-86a2-396e32fff7ca-kube-api-access-dgbhh\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.140537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.140515    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.140569 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.140545    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.241781 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.241746    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.241781 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.241796    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7221740b-e003-4af1-86a2-396e32fff7ca-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.242035 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.241811    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.242035 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.241858    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7221740b-e003-4af1-86a2-396e32fff7ca-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.242035 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.241913    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7221740b-e003-4af1-86a2-396e32fff7ca-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.242035 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.241937    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.242035 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.241958    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.242035 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.242015    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgbhh\" (UniqueName: \"kubernetes.io/projected/7221740b-e003-4af1-86a2-396e32fff7ca-kube-api-access-dgbhh\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.242447 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.242048    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.242447 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.242343    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.242560 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.242474    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.242560 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.242503    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.242663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.242555    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.242663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.242648    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7221740b-e003-4af1-86a2-396e32fff7ca-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.244683 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.244650    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.246083 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.246055    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7221740b-e003-4af1-86a2-396e32fff7ca-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.263727 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.263708    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7221740b-e003-4af1-86a2-396e32fff7ca-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.264801 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.264785    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgbhh\" (UniqueName: \"kubernetes.io/projected/7221740b-e003-4af1-86a2-396e32fff7ca-kube-api-access-dgbhh\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.272104 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.272070    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:37.364298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.364216    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"]
Oct 30 22:16:37.369609 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.369587    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.382167 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.382146    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:37.390625 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.390573    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-2b1e88-openshift-default-dockercfg-ljrwv\""
Oct 30 22:16:37.392798 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.392765    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"]
Oct 30 22:16:37.443057 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.443031    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.443202 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.443071    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.443202 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.443172    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.443516 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.443210    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137329f9-9437-46ab-b030-ead038ccd542-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.443516 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.443271    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.443516 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.443328    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.443516 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.443352    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137329f9-9437-46ab-b030-ead038ccd542-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.443516 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.443425    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8q27\" (UniqueName: \"kubernetes.io/projected/137329f9-9437-46ab-b030-ead038ccd542-kube-api-access-f8q27\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.443516 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.443462    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137329f9-9437-46ab-b030-ead038ccd542-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.460467 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.460443    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"]
Oct 30 22:16:37.461301 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:16:37.461275    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e978e1_1585_4947_8975_fe20d4942bc5.slice/crio-d79f21e5b00a068fc9c0ff9fd2d8ea07db1d3b9debb0b8242f81ce9a328d55ba WatchSource:0}: Error finding container d79f21e5b00a068fc9c0ff9fd2d8ea07db1d3b9debb0b8242f81ce9a328d55ba: Status 404 returned error can't find the container with id d79f21e5b00a068fc9c0ff9fd2d8ea07db1d3b9debb0b8242f81ce9a328d55ba
Oct 30 22:16:37.463922 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.463889    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:16:37.464022 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.463950    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:16:37.464022 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.463982    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:16:37.464937 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.464915    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:37.544025 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.543997    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.544138 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.544039    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137329f9-9437-46ab-b030-ead038ccd542-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.544138 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.544120    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8q27\" (UniqueName: \"kubernetes.io/projected/137329f9-9437-46ab-b030-ead038ccd542-kube-api-access-f8q27\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.544211 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.544158    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137329f9-9437-46ab-b030-ead038ccd542-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.544211 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.544188    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.544295 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.544236    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.544328 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.544290    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.544392 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.544323    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137329f9-9437-46ab-b030-ead038ccd542-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.544392 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.544352    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.544744 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.544726    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.544967 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.544952    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.545601 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.545521    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.545797 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.545780    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.546048 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.546029    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137329f9-9437-46ab-b030-ead038ccd542-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.547507 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.547483    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137329f9-9437-46ab-b030-ead038ccd542-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.547951 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.547924    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.566529 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.566437    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8q27\" (UniqueName: \"kubernetes.io/projected/137329f9-9437-46ab-b030-ead038ccd542-kube-api-access-f8q27\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.566529 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.566473    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137329f9-9437-46ab-b030-ead038ccd542-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.668983 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.668906    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:37.687200 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.686963    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:37.729865 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.726145    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"]
Oct 30 22:16:37.734739 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.734700    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:16:37.734863 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.734789    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:16:37.734863 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.734832    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:16:37.863988 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:37.863896    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:38.036139 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:38.036106    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"]
Oct 30 22:16:38.037192 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:16:38.037164    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137329f9_9437_46ab_b030_ead038ccd542.slice/crio-a1f7b76fc3bbed2d41d338873eb5d37bf25e8b6c1a8028c05601652c07aa7a10 WatchSource:0}: Error finding container a1f7b76fc3bbed2d41d338873eb5d37bf25e8b6c1a8028c05601652c07aa7a10: Status 404 returned error can't find the container with id a1f7b76fc3bbed2d41d338873eb5d37bf25e8b6c1a8028c05601652c07aa7a10
Oct 30 22:16:38.039879 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:38.039849    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:16:38.039961 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:38.039912    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:16:38.039961 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:38.039938    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:16:38.335385 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:38.335288    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt" event={"ID":"137329f9-9437-46ab-b030-ead038ccd542","Type":"ContainerStarted","Data":"b95598dfcf28df436f7a2d55ed03f23337e74d6cf084ac26b9ceca13f16a5dc0"}
Oct 30 22:16:38.335385 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:38.335358    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt" event={"ID":"137329f9-9437-46ab-b030-ead038ccd542","Type":"ContainerStarted","Data":"a1f7b76fc3bbed2d41d338873eb5d37bf25e8b6c1a8028c05601652c07aa7a10"}
Oct 30 22:16:38.337253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:38.337206    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg" event={"ID":"7221740b-e003-4af1-86a2-396e32fff7ca","Type":"ContainerStarted","Data":"d385f1fd17d203664f64a9a612f0007617ba893cef05ea957423d7e0b3cc2f95"}
Oct 30 22:16:38.337390 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:38.337282    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg" event={"ID":"7221740b-e003-4af1-86a2-396e32fff7ca","Type":"ContainerStarted","Data":"ea4bbe3ee57db79ddd8e0b2bd81f388b4acac2ee5c6ed42846a3e2bdb01b651c"}
Oct 30 22:16:38.338941 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:38.338918    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t" event={"ID":"32e978e1-1585-4947-8975-fe20d4942bc5","Type":"ContainerStarted","Data":"c7bcd4f15b101168d04b0b906613054c5076a4441a0a6202a6bdec339694c1f1"}
Oct 30 22:16:38.339043 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:38.338960    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t" event={"ID":"32e978e1-1585-4947-8975-fe20d4942bc5","Type":"ContainerStarted","Data":"d79f21e5b00a068fc9c0ff9fd2d8ea07db1d3b9debb0b8242f81ce9a328d55ba"}
Oct 30 22:16:38.379884 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:38.379835    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt" podStartSLOduration=1.379818724 podStartE2EDuration="1.379818724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:38.37757624 +0000 UTC m=+583.733942565" watchObservedRunningTime="2026-10-30 22:16:38.379818724 +0000 UTC m=+583.736185037"
Oct 30 22:16:38.383195 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:38.383176    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:38.388781 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:38.388757    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:38.423616 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:38.423573    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg" podStartSLOduration=1.423560275 podStartE2EDuration="1.423560275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:38.421925066 +0000 UTC m=+583.778291380" watchObservedRunningTime="2026-10-30 22:16:38.423560275 +0000 UTC m=+583.779926586"
Oct 30 22:16:38.468021 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:38.467973    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t" podStartSLOduration=2.467961626 podStartE2EDuration="2.467961626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:38.466839621 +0000 UTC m=+583.823205936" watchObservedRunningTime="2026-10-30 22:16:38.467961626 +0000 UTC m=+583.824327941"
Oct 30 22:16:38.687435 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:38.687366    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:39.140859 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:39.140826    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:39.146008 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:39.145984    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:39.343357 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:39.343323    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:39.343634 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:39.343605    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:39.344468 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:39.344448    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:16:39.344588 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:39.344565    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:16:39.692666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:39.692633    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:40.300857 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:16:40.300826    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:16:40.348488 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:40.348445    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:40.349626 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:40.349607    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:16:40.360779 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:40.360742    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:40.534538 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:40.534505    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:41.085792 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:41.085755    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:41.310599 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:41.310550    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:41.568192 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:41.568163    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:41.922484 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:41.922351    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:42.140747 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:42.140712    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:42.345241 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:42.345196    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:42.878722 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:42.878683    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:43.024486 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:43.024441    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48"]
Oct 30 22:16:43.037802 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:43.037774    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48"
Oct 30 22:16:43.050939 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:43.050915    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48"]
Oct 30 22:16:43.060517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:43.060498    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-09d0d5\""
Oct 30 22:16:43.060774 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:43.060759    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-09d0d5-dockercfg-qqm4s\""
Oct 30 22:16:43.084270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:43.084218    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:43.118861 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:43.117837    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm94j\" (UniqueName: \"kubernetes.io/projected/51bd7538-efdc-40b3-b43c-35e7714a1014-kube-api-access-cm94j\") pod \"payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48\" (UID: \"51bd7538-efdc-40b3-b43c-35e7714a1014\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48"
Oct 30 22:16:43.119193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:43.119158    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51bd7538-efdc-40b3-b43c-35e7714a1014-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48\" (UID: \"51bd7538-efdc-40b3-b43c-35e7714a1014\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48"
Oct 30 22:16:43.122840 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:43.122801    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb"]
Oct 30 22:16:43.129764 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:43.129700    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb"
Oct 30 22:16:43.150385 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:43.149932    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb"]
Oct 30 22:16:43.220098 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:43.220069    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51bd7538-efdc-40b3-b43c-35e7714a1014-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48\" (UID: \"51bd7538-efdc-40b3-b43c-35e7714a1014\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48"
Oct 30 22:16:43.220298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:43.220105    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grjtr\" (UniqueName: \"kubernetes.io/projected/6d033fb5-f7fb-4480-9cf6-57effbff1402-kube-api-access-grjtr\") pod \"payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb\" (UID: \"6d033fb5-f7fb-4480-9cf6-57effbff1402\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb"
Oct 30 22:16:43.220298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:43.220191    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d033fb5-f7fb-4480-9cf6-57effbff1402-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb\" (UID: \"6d033fb5-f7fb-4480-9cf6-57effbff1402\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb"
Oct 30 22:16:43.220298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:43.220236    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm94j\" (UniqueName: \"kubernetes.io/projected/51bd7538-efdc-40b3-b43c-35e7714a1014-kube-api-access-cm94j\") pod \"payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48\" (UID: \"51bd7538-efdc-40b3-b43c-35e7714a1014\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48"
Oct 30 22:16:43.220842 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:43.220818    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51bd7538-efdc-40b3-b43c-35e7714a1014-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48\" (UID: \"51bd7538-efdc-40b3-b43c-35e7714a1014\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48"
Oct 30 22:16:43.239549 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:43.239523    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm94j\" (UniqueName: \"kubernetes.io/projected/51bd7538-efdc-40b3-b43c-35e7714a1014-kube-api-access-cm94j\") pod \"payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48\" (UID: \"51bd7538-efdc-40b3-b43c-35e7714a1014\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48"
Oct 30 22:16:43.321651 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:43.321619    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d033fb5-f7fb-4480-9cf6-57effbff1402-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb\" (UID: \"6d033fb5-f7fb-4480-9cf6-57effbff1402\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb"
Oct 30 22:16:43.321830 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:43.321687    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grjtr\" (UniqueName: \"kubernetes.io/projected/6d033fb5-f7fb-4480-9cf6-57effbff1402-kube-api-access-grjtr\") pod \"payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb\" (UID: \"6d033fb5-f7fb-4480-9cf6-57effbff1402\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb"
Oct 30 22:16:43.322511 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:43.322491    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d033fb5-f7fb-4480-9cf6-57effbff1402-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb\" (UID: \"6d033fb5-f7fb-4480-9cf6-57effbff1402\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb"
Oct 30 22:16:43.341548 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:43.341518    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grjtr\" (UniqueName: \"kubernetes.io/projected/6d033fb5-f7fb-4480-9cf6-57effbff1402-kube-api-access-grjtr\") pod \"payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb\" (UID: \"6d033fb5-f7fb-4480-9cf6-57effbff1402\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb"
Oct 30 22:16:43.351191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:43.351171    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48"
Oct 30 22:16:43.454715 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:43.454684    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb"
Oct 30 22:16:43.652188 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:43.652159    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48"]
Oct 30 22:16:43.652583 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:16:43.652562    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51bd7538_efdc_40b3_b43c_35e7714a1014.slice/crio-cdde1b2dc9d180d09ce553561fda22db61ad8d83282f206a4570b20810793481 WatchSource:0}: Error finding container cdde1b2dc9d180d09ce553561fda22db61ad8d83282f206a4570b20810793481: Status 404 returned error can't find the container with id cdde1b2dc9d180d09ce553561fda22db61ad8d83282f206a4570b20810793481
Oct 30 22:16:43.761401 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:43.761371    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb"]
Oct 30 22:16:43.761642 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:16:43.761622    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d033fb5_f7fb_4480_9cf6_57effbff1402.slice/crio-49d90ca08d9dc563bf71f8d0862d3e3805712f4acff0b85e2d59ef002da3be77 WatchSource:0}: Error finding container 49d90ca08d9dc563bf71f8d0862d3e3805712f4acff0b85e2d59ef002da3be77: Status 404 returned error can't find the container with id 49d90ca08d9dc563bf71f8d0862d3e3805712f4acff0b85e2d59ef002da3be77
Oct 30 22:16:44.377360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:44.377320    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb" event={"ID":"6d033fb5-f7fb-4480-9cf6-57effbff1402","Type":"ContainerStarted","Data":"0fcf7dfcbcdad3634ad5e3b42330135b9db8e1bb7855f274d513b8e4306747ab"}
Oct 30 22:16:44.377781 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:44.377384    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb" event={"ID":"6d033fb5-f7fb-4480-9cf6-57effbff1402","Type":"ContainerStarted","Data":"49d90ca08d9dc563bf71f8d0862d3e3805712f4acff0b85e2d59ef002da3be77"}
Oct 30 22:16:44.377781 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:44.377529    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb"
Oct 30 22:16:44.379139 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:44.379111    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48" event={"ID":"51bd7538-efdc-40b3-b43c-35e7714a1014","Type":"ContainerStarted","Data":"978f57ae7cfbaf7ea8ec4b8953ba32fb54ef7f7fc4e3f3cef67ed1207689e463"}
Oct 30 22:16:44.379280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:44.379161    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48" event={"ID":"51bd7538-efdc-40b3-b43c-35e7714a1014","Type":"ContainerStarted","Data":"cdde1b2dc9d180d09ce553561fda22db61ad8d83282f206a4570b20810793481"}
Oct 30 22:16:44.379280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:44.379190    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48"
Oct 30 22:16:44.417716 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:44.417666    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb" podStartSLOduration=1.417654531 podStartE2EDuration="1.417654531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:44.416256875 +0000 UTC m=+589.772623189" watchObservedRunningTime="2026-10-30 22:16:44.417654531 +0000 UTC m=+589.774020845"
Oct 30 22:16:44.425867 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:44.425844    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:44.457572 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:44.457530    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48" podStartSLOduration=2.457517017 podStartE2EDuration="2.457517017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:44.455587815 +0000 UTC m=+589.811954251" watchObservedRunningTime="2026-10-30 22:16:44.457517017 +0000 UTC m=+589.813883331"
Oct 30 22:16:44.618616 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:44.613181    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:45.219089 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.219051    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:45.340914 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.340874    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx"]
Oct 30 22:16:45.344990 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.344966    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx"
Oct 30 22:16:45.354712 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.354683    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx"]
Oct 30 22:16:45.364697 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.364672    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-d52c59-dockercfg-kbf6h\""
Oct 30 22:16:45.387273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.369802    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-d52c59\""
Oct 30 22:16:45.390643 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.390615    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:45.443722 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.443683    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z79gk\" (UniqueName: \"kubernetes.io/projected/5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0-kube-api-access-z79gk\") pod \"payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx\" (UID: \"5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx"
Oct 30 22:16:45.444788 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.444760    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx\" (UID: \"5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx"
Oct 30 22:16:45.546349 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.546313    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx\" (UID: \"5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx"
Oct 30 22:16:45.546564 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.546384    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z79gk\" (UniqueName: \"kubernetes.io/projected/5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0-kube-api-access-z79gk\") pod \"payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx\" (UID: \"5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx"
Oct 30 22:16:45.547376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.547352    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx\" (UID: \"5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx"
Oct 30 22:16:45.567380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.567351    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z79gk\" (UniqueName: \"kubernetes.io/projected/5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0-kube-api-access-z79gk\") pod \"payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx\" (UID: \"5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx"
Oct 30 22:16:45.579458 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.579429    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:45.635885 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.635851    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5"]
Oct 30 22:16:45.639641 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.639624    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5"
Oct 30 22:16:45.657743 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.657717    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-421583-serving-cert\""
Oct 30 22:16:45.657865 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.657756    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5"]
Oct 30 22:16:45.662831 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.662809    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx"
Oct 30 22:16:45.748115 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.748059    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-maas-api-tls\") pod \"maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5\" (UID: \"cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5"
Oct 30 22:16:45.748305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.748194    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-metrics-certs\") pod \"maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5\" (UID: \"cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5"
Oct 30 22:16:45.748381 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.748357    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5\" (UID: \"cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5"
Oct 30 22:16:45.748658 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.748442    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2nwt\" (UniqueName: \"kubernetes.io/projected/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-kube-api-access-x2nwt\") pod \"maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5\" (UID: \"cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5"
Oct 30 22:16:45.787129 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.787096    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:45.849867 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.849779    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-maas-api-tls\") pod \"maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5\" (UID: \"cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5"
Oct 30 22:16:45.849867 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.849832    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-metrics-certs\") pod \"maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5\" (UID: \"cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5"
Oct 30 22:16:45.850081 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.849874    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5\" (UID: \"cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5"
Oct 30 22:16:45.850081 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.849902    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2nwt\" (UniqueName: \"kubernetes.io/projected/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-kube-api-access-x2nwt\") pod \"maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5\" (UID: \"cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5"
Oct 30 22:16:45.850873 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.850848    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5\" (UID: \"cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5"
Oct 30 22:16:45.852447 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.852427    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-maas-api-tls\") pod \"maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5\" (UID: \"cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5"
Oct 30 22:16:45.852611 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.852590    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-metrics-certs\") pod \"maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5\" (UID: \"cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5"
Oct 30 22:16:45.869046 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.869024    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2nwt\" (UniqueName: \"kubernetes.io/projected/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-kube-api-access-x2nwt\") pod \"maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5\" (UID: \"cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5"
Oct 30 22:16:45.949332 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.949301    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:45.952826 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.952802    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5"
Oct 30 22:16:45.976858 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:45.976834    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx"]
Oct 30 22:16:45.977017 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:16:45.976999    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb47a7e_8385_4154_a3b2_8ff2e1ae39b0.slice/crio-9c94dbc84b755913122d7ea5607de5e1f04f9dd81e0ed201de497d255b9d5a10 WatchSource:0}: Error finding container 9c94dbc84b755913122d7ea5607de5e1f04f9dd81e0ed201de497d255b9d5a10: Status 404 returned error can't find the container with id 9c94dbc84b755913122d7ea5607de5e1f04f9dd81e0ed201de497d255b9d5a10
Oct 30 22:16:46.272015 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:46.271987    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5"]
Oct 30 22:16:46.273114 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:16:46.273077    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb1b2e2_ed49_4662_b1e0_07d1e1cce6ad.slice/crio-e81c61fa4c566680639181e46b4616dd1ce42466a7f6ca4fa00315ccb8fba6bf WatchSource:0}: Error finding container e81c61fa4c566680639181e46b4616dd1ce42466a7f6ca4fa00315ccb8fba6bf: Status 404 returned error can't find the container with id e81c61fa4c566680639181e46b4616dd1ce42466a7f6ca4fa00315ccb8fba6bf
Oct 30 22:16:46.398808 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:46.398769    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5" event={"ID":"cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad","Type":"ContainerStarted","Data":"e81c61fa4c566680639181e46b4616dd1ce42466a7f6ca4fa00315ccb8fba6bf"}
Oct 30 22:16:46.400439 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:46.400414    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx" event={"ID":"5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0","Type":"ContainerStarted","Data":"1ca05f1cdfe7127647219bc1e430016dede4b8acfd227ad52c48c37c5e3fce39"}
Oct 30 22:16:46.400551 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:46.400466    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx"
Oct 30 22:16:46.400551 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:46.400490    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx" event={"ID":"5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0","Type":"ContainerStarted","Data":"9c94dbc84b755913122d7ea5607de5e1f04f9dd81e0ed201de497d255b9d5a10"}
Oct 30 22:16:46.439914 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:46.439863    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx" podStartSLOduration=1.439849765 podStartE2EDuration="1.439849765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:46.438628585 +0000 UTC m=+591.794994900" watchObservedRunningTime="2026-10-30 22:16:46.439849765 +0000 UTC m=+591.796216084"
Oct 30 22:16:46.678697 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:46.678653    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:47.408178 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.408140    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5" event={"ID":"cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad","Type":"ContainerStarted","Data":"c1538ffd80e4348aeb7b5a85fad0c1188983302480493c923d35b05137a4d21c"}
Oct 30 22:16:47.408658 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.408337    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5"
Oct 30 22:16:47.450048 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.449998    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5" podStartSLOduration=2.128943504 podStartE2EDuration="2.449985222s" totalImagesPullingTime="321.041718ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:47.448986461 +0000 UTC m=+592.805352773" watchObservedRunningTime="2026-10-30 22:16:47.449985222 +0000 UTC m=+592.806351534"
Oct 30 22:16:47.475372 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.475342    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6"]
Oct 30 22:16:47.479007 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.478991    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6"
Oct 30 22:16:47.494470 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.494446    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-012ba0\""
Oct 30 22:16:47.496171 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.496145    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6"]
Oct 30 22:16:47.554015 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.553982    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc"]
Oct 30 22:16:47.557620 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.557605    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc"
Oct 30 22:16:47.568552 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.568529    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzq9p\" (UniqueName: \"kubernetes.io/projected/b812174d-906c-487b-91a9-7a1620aca44e-kube-api-access-zzq9p\") pod \"payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6\" (UID: \"b812174d-906c-487b-91a9-7a1620aca44e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6"
Oct 30 22:16:47.568739 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.568699    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b812174d-906c-487b-91a9-7a1620aca44e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6\" (UID: \"b812174d-906c-487b-91a9-7a1620aca44e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6"
Oct 30 22:16:47.573004 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.572983    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-012ba0-dockercfg-kb99n\""
Oct 30 22:16:47.574519 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.574496    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc"]
Oct 30 22:16:47.618405 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.618380    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6"]
Oct 30 22:16:47.622293 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.622273    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6"
Oct 30 22:16:47.640669 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.640639    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6"]
Oct 30 22:16:47.669892 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.669817    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44c69100-209c-49c3-9c84-cb2d36028de5-metrics-certs\") pod \"maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6\" (UID: \"44c69100-209c-49c3-9c84-cb2d36028de5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6"
Oct 30 22:16:47.669892 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.669875    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e0a2179-dbbd-49c3-b6af-90aec117a44c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc\" (UID: \"2e0a2179-dbbd-49c3-b6af-90aec117a44c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc"
Oct 30 22:16:47.670075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.669957    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n46hf\" (UniqueName: \"kubernetes.io/projected/44c69100-209c-49c3-9c84-cb2d36028de5-kube-api-access-n46hf\") pod \"maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6\" (UID: \"44c69100-209c-49c3-9c84-cb2d36028de5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6"
Oct 30 22:16:47.670125 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.670071    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b812174d-906c-487b-91a9-7a1620aca44e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6\" (UID: \"b812174d-906c-487b-91a9-7a1620aca44e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6"
Oct 30 22:16:47.670125 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.670114    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44c69100-209c-49c3-9c84-cb2d36028de5-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6\" (UID: \"44c69100-209c-49c3-9c84-cb2d36028de5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6"
Oct 30 22:16:47.670209 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.670164    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfcv8\" (UniqueName: \"kubernetes.io/projected/2e0a2179-dbbd-49c3-b6af-90aec117a44c-kube-api-access-qfcv8\") pod \"payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc\" (UID: \"2e0a2179-dbbd-49c3-b6af-90aec117a44c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc"
Oct 30 22:16:47.670209 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.670188    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzq9p\" (UniqueName: \"kubernetes.io/projected/b812174d-906c-487b-91a9-7a1620aca44e-kube-api-access-zzq9p\") pod \"payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6\" (UID: \"b812174d-906c-487b-91a9-7a1620aca44e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6"
Oct 30 22:16:47.670325 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.670256    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44c69100-209c-49c3-9c84-cb2d36028de5-maas-api-tls\") pod \"maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6\" (UID: \"44c69100-209c-49c3-9c84-cb2d36028de5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6"
Oct 30 22:16:47.670803 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.670781    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b812174d-906c-487b-91a9-7a1620aca44e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6\" (UID: \"b812174d-906c-487b-91a9-7a1620aca44e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6"
Oct 30 22:16:47.673457 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.673441    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-09d0d5-serving-cert\""
Oct 30 22:16:47.689456 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.689437    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzq9p\" (UniqueName: \"kubernetes.io/projected/b812174d-906c-487b-91a9-7a1620aca44e-kube-api-access-zzq9p\") pod \"payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6\" (UID: \"b812174d-906c-487b-91a9-7a1620aca44e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6"
Oct 30 22:16:47.771692 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.771663    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44c69100-209c-49c3-9c84-cb2d36028de5-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6\" (UID: \"44c69100-209c-49c3-9c84-cb2d36028de5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6"
Oct 30 22:16:47.771838 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.771716    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfcv8\" (UniqueName: \"kubernetes.io/projected/2e0a2179-dbbd-49c3-b6af-90aec117a44c-kube-api-access-qfcv8\") pod \"payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc\" (UID: \"2e0a2179-dbbd-49c3-b6af-90aec117a44c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc"
Oct 30 22:16:47.771838 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.771738    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44c69100-209c-49c3-9c84-cb2d36028de5-maas-api-tls\") pod \"maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6\" (UID: \"44c69100-209c-49c3-9c84-cb2d36028de5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6"
Oct 30 22:16:47.771838 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.771764    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44c69100-209c-49c3-9c84-cb2d36028de5-metrics-certs\") pod \"maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6\" (UID: \"44c69100-209c-49c3-9c84-cb2d36028de5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6"
Oct 30 22:16:47.771838 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.771804    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e0a2179-dbbd-49c3-b6af-90aec117a44c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc\" (UID: \"2e0a2179-dbbd-49c3-b6af-90aec117a44c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc"
Oct 30 22:16:47.771838 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.771820    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n46hf\" (UniqueName: \"kubernetes.io/projected/44c69100-209c-49c3-9c84-cb2d36028de5-kube-api-access-n46hf\") pod \"maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6\" (UID: \"44c69100-209c-49c3-9c84-cb2d36028de5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6"
Oct 30 22:16:47.772507 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.772405    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44c69100-209c-49c3-9c84-cb2d36028de5-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6\" (UID: \"44c69100-209c-49c3-9c84-cb2d36028de5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6"
Oct 30 22:16:47.772704 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.772571    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e0a2179-dbbd-49c3-b6af-90aec117a44c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc\" (UID: \"2e0a2179-dbbd-49c3-b6af-90aec117a44c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc"
Oct 30 22:16:47.774282 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.774262    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44c69100-209c-49c3-9c84-cb2d36028de5-metrics-certs\") pod \"maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6\" (UID: \"44c69100-209c-49c3-9c84-cb2d36028de5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6"
Oct 30 22:16:47.774403 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.774386    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44c69100-209c-49c3-9c84-cb2d36028de5-maas-api-tls\") pod \"maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6\" (UID: \"44c69100-209c-49c3-9c84-cb2d36028de5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6"
Oct 30 22:16:47.790872 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.790845    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n46hf\" (UniqueName: \"kubernetes.io/projected/44c69100-209c-49c3-9c84-cb2d36028de5-kube-api-access-n46hf\") pod \"maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6\" (UID: \"44c69100-209c-49c3-9c84-cb2d36028de5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6"
Oct 30 22:16:47.790964 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.790903    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfcv8\" (UniqueName: \"kubernetes.io/projected/2e0a2179-dbbd-49c3-b6af-90aec117a44c-kube-api-access-qfcv8\") pod \"payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc\" (UID: \"2e0a2179-dbbd-49c3-b6af-90aec117a44c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc"
Oct 30 22:16:47.791591 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.791570    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6"
Oct 30 22:16:47.882804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.874049    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc"
Oct 30 22:16:47.938741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:47.935871    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6"
Oct 30 22:16:48.092788 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:48.092763    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6"]
Oct 30 22:16:48.092954 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:16:48.092928    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb812174d_906c_487b_91a9_7a1620aca44e.slice/crio-b5afa9711a2ccc2f84d35e8a782a83d3d5cd94dbead18a010725f949089f0ff8 WatchSource:0}: Error finding container b5afa9711a2ccc2f84d35e8a782a83d3d5cd94dbead18a010725f949089f0ff8: Status 404 returned error can't find the container with id b5afa9711a2ccc2f84d35e8a782a83d3d5cd94dbead18a010725f949089f0ff8
Oct 30 22:16:48.188061 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:48.188021    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:48.234210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:48.234187    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc"]
Oct 30 22:16:48.234905 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:16:48.234884    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e0a2179_dbbd_49c3_b6af_90aec117a44c.slice/crio-a31da173b660dfee9e31616f8bf439b90351997fcc017288b913729605519cab WatchSource:0}: Error finding container a31da173b660dfee9e31616f8bf439b90351997fcc017288b913729605519cab: Status 404 returned error can't find the container with id a31da173b660dfee9e31616f8bf439b90351997fcc017288b913729605519cab
Oct 30 22:16:48.246949 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:16:48.246921    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c69100_209c_49c3_9c84_cb2d36028de5.slice/crio-9b0d183fbc345efe19d430ad224ca6c50c31a7d547770759160e02563aaf00cc WatchSource:0}: Error finding container 9b0d183fbc345efe19d430ad224ca6c50c31a7d547770759160e02563aaf00cc: Status 404 returned error can't find the container with id 9b0d183fbc345efe19d430ad224ca6c50c31a7d547770759160e02563aaf00cc
Oct 30 22:16:48.247789 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:48.247767    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6"]
Oct 30 22:16:48.415310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:48.415275    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6" event={"ID":"44c69100-209c-49c3-9c84-cb2d36028de5","Type":"ContainerStarted","Data":"9b0d183fbc345efe19d430ad224ca6c50c31a7d547770759160e02563aaf00cc"}
Oct 30 22:16:48.416842 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:48.416818    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc" event={"ID":"2e0a2179-dbbd-49c3-b6af-90aec117a44c","Type":"ContainerStarted","Data":"dcf269360f829facc20b88c5d510c5292f50e8a0690a69f665911fb548b1f89e"}
Oct 30 22:16:48.416955 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:48.416866    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc" event={"ID":"2e0a2179-dbbd-49c3-b6af-90aec117a44c","Type":"ContainerStarted","Data":"a31da173b660dfee9e31616f8bf439b90351997fcc017288b913729605519cab"}
Oct 30 22:16:48.417006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:48.416992    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc"
Oct 30 22:16:48.418552 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:48.418532    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6" event={"ID":"b812174d-906c-487b-91a9-7a1620aca44e","Type":"ContainerStarted","Data":"632093beb73e141eed51a24f65cd85fb6454fb25f32d92a7100dee5e129bbbb0"}
Oct 30 22:16:48.418637 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:48.418569    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6" event={"ID":"b812174d-906c-487b-91a9-7a1620aca44e","Type":"ContainerStarted","Data":"b5afa9711a2ccc2f84d35e8a782a83d3d5cd94dbead18a010725f949089f0ff8"}
Oct 30 22:16:48.456530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:48.456477    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc" podStartSLOduration=1.456462788 podStartE2EDuration="1.456462788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:48.456051149 +0000 UTC m=+593.812417464" watchObservedRunningTime="2026-10-30 22:16:48.456462788 +0000 UTC m=+593.812829105"
Oct 30 22:16:48.498537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:48.498492    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6" podStartSLOduration=1.498479454 podStartE2EDuration="1.498479454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:48.497019266 +0000 UTC m=+593.853385580" watchObservedRunningTime="2026-10-30 22:16:48.498479454 +0000 UTC m=+593.854845767"
Oct 30 22:16:49.426000 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:49.425963    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6" event={"ID":"44c69100-209c-49c3-9c84-cb2d36028de5","Type":"ContainerStarted","Data":"a213e973df03acd4bc29a61a6c8adbeac2350142409f42bdea7a7ad902cc44fe"}
Oct 30 22:16:49.426477 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:49.426301    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6"
Oct 30 22:16:49.426477 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:49.426345    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6"
Oct 30 22:16:49.476566 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:49.476515    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6" podStartSLOduration=2.062073252 podStartE2EDuration="2.476501833s" totalImagesPullingTime="414.428581ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:49.476188573 +0000 UTC m=+594.832554888" watchObservedRunningTime="2026-10-30 22:16:49.476501833 +0000 UTC m=+594.832868146"
Oct 30 22:16:49.592675 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:49.592638    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:49.791716 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:49.791675    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:49.967414 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:49.967379    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:50.469411 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:50.469379    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:50.641787 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:50.641748    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:51.522102 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:51.522071    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7"]
Oct 30 22:16:51.526166 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:51.526146    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7"
Oct 30 22:16:51.554164 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:51.554137    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-54114e-serving-cert\""
Oct 30 22:16:51.556622 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:51.556597    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7"]
Oct 30 22:16:51.626692 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:51.626655    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxqq9\" (UniqueName: \"kubernetes.io/projected/ddfac372-f0f0-4e7b-a661-007adc297ebd-kube-api-access-sxqq9\") pod \"maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7\" (UID: \"ddfac372-f0f0-4e7b-a661-007adc297ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7"
Oct 30 22:16:51.626866 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:51.626704    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddfac372-f0f0-4e7b-a661-007adc297ebd-metrics-certs\") pod \"maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7\" (UID: \"ddfac372-f0f0-4e7b-a661-007adc297ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7"
Oct 30 22:16:51.626866 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:51.626855    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddfac372-f0f0-4e7b-a661-007adc297ebd-maas-api-tls\") pod \"maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7\" (UID: \"ddfac372-f0f0-4e7b-a661-007adc297ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7"
Oct 30 22:16:51.626951 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:51.626880    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddfac372-f0f0-4e7b-a661-007adc297ebd-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7\" (UID: \"ddfac372-f0f0-4e7b-a661-007adc297ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7"
Oct 30 22:16:51.701604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:51.701570    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:51.728040 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:51.728014    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddfac372-f0f0-4e7b-a661-007adc297ebd-maas-api-tls\") pod \"maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7\" (UID: \"ddfac372-f0f0-4e7b-a661-007adc297ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7"
Oct 30 22:16:51.728324 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:51.728050    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddfac372-f0f0-4e7b-a661-007adc297ebd-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7\" (UID: \"ddfac372-f0f0-4e7b-a661-007adc297ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7"
Oct 30 22:16:51.728324 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:51.728104    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxqq9\" (UniqueName: \"kubernetes.io/projected/ddfac372-f0f0-4e7b-a661-007adc297ebd-kube-api-access-sxqq9\") pod \"maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7\" (UID: \"ddfac372-f0f0-4e7b-a661-007adc297ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7"
Oct 30 22:16:51.728324 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:51.728148    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddfac372-f0f0-4e7b-a661-007adc297ebd-metrics-certs\") pod \"maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7\" (UID: \"ddfac372-f0f0-4e7b-a661-007adc297ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7"
Oct 30 22:16:51.729010 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:51.728947    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddfac372-f0f0-4e7b-a661-007adc297ebd-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7\" (UID: \"ddfac372-f0f0-4e7b-a661-007adc297ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7"
Oct 30 22:16:51.730870 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:51.730846    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddfac372-f0f0-4e7b-a661-007adc297ebd-metrics-certs\") pod \"maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7\" (UID: \"ddfac372-f0f0-4e7b-a661-007adc297ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7"
Oct 30 22:16:51.730955 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:51.730924    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddfac372-f0f0-4e7b-a661-007adc297ebd-maas-api-tls\") pod \"maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7\" (UID: \"ddfac372-f0f0-4e7b-a661-007adc297ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7"
Oct 30 22:16:51.751778 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:51.751758    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxqq9\" (UniqueName: \"kubernetes.io/projected/ddfac372-f0f0-4e7b-a661-007adc297ebd-kube-api-access-sxqq9\") pod \"maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7\" (UID: \"ddfac372-f0f0-4e7b-a661-007adc297ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7"
Oct 30 22:16:51.838322 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:51.838238    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7"
Oct 30 22:16:51.860557 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:51.860527    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:52.139003 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:52.138972    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7"]
Oct 30 22:16:52.139139 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:16:52.139043    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddfac372_f0f0_4e7b_a661_007adc297ebd.slice/crio-1c1e1112f4397bec5f95c4db1dd987e682fc97e45466fd5726dc229a507c16e1 WatchSource:0}: Error finding container 1c1e1112f4397bec5f95c4db1dd987e682fc97e45466fd5726dc229a507c16e1: Status 404 returned error can't find the container with id 1c1e1112f4397bec5f95c4db1dd987e682fc97e45466fd5726dc229a507c16e1
Oct 30 22:16:52.445492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:52.445400    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7" event={"ID":"ddfac372-f0f0-4e7b-a661-007adc297ebd","Type":"ContainerStarted","Data":"1c1e1112f4397bec5f95c4db1dd987e682fc97e45466fd5726dc229a507c16e1"}
Oct 30 22:16:53.381498 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:53.381468    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:53.430491 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:53.430461    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5"
Oct 30 22:16:53.452101 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:53.452071    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7" event={"ID":"ddfac372-f0f0-4e7b-a661-007adc297ebd","Type":"ContainerStarted","Data":"f5811c9e77f137eae0830817988c7e9c3107c3c8832b287e9112cecc96c2f157"}
Oct 30 22:16:53.452336 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:53.452130    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7"
Oct 30 22:16:53.516294 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:53.516241    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7" podStartSLOduration=2.16448504 podStartE2EDuration="2.516209791s" totalImagesPullingTime="351.724751ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:53.514715703 +0000 UTC m=+598.871082019" watchObservedRunningTime="2026-10-30 22:16:53.516209791 +0000 UTC m=+598.872576104"
Oct 30 22:16:53.555771 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:53.555727    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:53.722176 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:53.722089    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:54.865885 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:54.865771    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 22:16:54.870954 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:54.870921    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 22:16:55.160219 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:55.160122    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:55.394766 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:55.394741    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48"
Oct 30 22:16:55.394940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:55.394812    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb"
Oct 30 22:16:55.438261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:55.438169    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6"
Oct 30 22:16:56.850424 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:56.850387    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228"]
Oct 30 22:16:56.855631 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:56.855613    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228"
Oct 30 22:16:56.875804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:56.875768    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228"]
Oct 30 22:16:56.895070 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:56.895041    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-0ad7c5-serving-cert\""
Oct 30 22:16:56.982846 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:56.982811    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79gb4\" (UniqueName: \"kubernetes.io/projected/aa389a05-73a7-490b-8d2d-a8e34d82b0af-kube-api-access-79gb4\") pod \"maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228\" (UID: \"aa389a05-73a7-490b-8d2d-a8e34d82b0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228"
Oct 30 22:16:56.983004 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:56.982875    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa389a05-73a7-490b-8d2d-a8e34d82b0af-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228\" (UID: \"aa389a05-73a7-490b-8d2d-a8e34d82b0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228"
Oct 30 22:16:56.983077 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:56.983039    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa389a05-73a7-490b-8d2d-a8e34d82b0af-maas-api-tls\") pod \"maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228\" (UID: \"aa389a05-73a7-490b-8d2d-a8e34d82b0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228"
Oct 30 22:16:56.983125 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:56.983077    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa389a05-73a7-490b-8d2d-a8e34d82b0af-metrics-certs\") pod \"maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228\" (UID: \"aa389a05-73a7-490b-8d2d-a8e34d82b0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228"
Oct 30 22:16:57.035648 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:57.035617    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:57.083731 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:57.083698    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa389a05-73a7-490b-8d2d-a8e34d82b0af-maas-api-tls\") pod \"maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228\" (UID: \"aa389a05-73a7-490b-8d2d-a8e34d82b0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228"
Oct 30 22:16:57.083903 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:57.083744    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa389a05-73a7-490b-8d2d-a8e34d82b0af-metrics-certs\") pod \"maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228\" (UID: \"aa389a05-73a7-490b-8d2d-a8e34d82b0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228"
Oct 30 22:16:57.083903 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:57.083796    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79gb4\" (UniqueName: \"kubernetes.io/projected/aa389a05-73a7-490b-8d2d-a8e34d82b0af-kube-api-access-79gb4\") pod \"maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228\" (UID: \"aa389a05-73a7-490b-8d2d-a8e34d82b0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228"
Oct 30 22:16:57.083903 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:57.083852    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa389a05-73a7-490b-8d2d-a8e34d82b0af-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228\" (UID: \"aa389a05-73a7-490b-8d2d-a8e34d82b0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228"
Oct 30 22:16:57.084592 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:57.084570    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa389a05-73a7-490b-8d2d-a8e34d82b0af-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228\" (UID: \"aa389a05-73a7-490b-8d2d-a8e34d82b0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228"
Oct 30 22:16:57.086317 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:57.086300    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa389a05-73a7-490b-8d2d-a8e34d82b0af-metrics-certs\") pod \"maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228\" (UID: \"aa389a05-73a7-490b-8d2d-a8e34d82b0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228"
Oct 30 22:16:57.086595 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:57.086562    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa389a05-73a7-490b-8d2d-a8e34d82b0af-maas-api-tls\") pod \"maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228\" (UID: \"aa389a05-73a7-490b-8d2d-a8e34d82b0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228"
Oct 30 22:16:57.103078 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:57.103028    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79gb4\" (UniqueName: \"kubernetes.io/projected/aa389a05-73a7-490b-8d2d-a8e34d82b0af-kube-api-access-79gb4\") pod \"maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228\" (UID: \"aa389a05-73a7-490b-8d2d-a8e34d82b0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228"
Oct 30 22:16:57.169279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:57.169241    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228"
Oct 30 22:16:57.208395 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:57.208348    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:57.435381 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:57.434642    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx"
Oct 30 22:16:57.506363 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:16:57.506336    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa389a05_73a7_490b_8d2d_a8e34d82b0af.slice/crio-84698186aaf6328aa2cd903f0a7d4ae5a0fffcd3dfa1d44aa2b80990370e3225 WatchSource:0}: Error finding container 84698186aaf6328aa2cd903f0a7d4ae5a0fffcd3dfa1d44aa2b80990370e3225: Status 404 returned error can't find the container with id 84698186aaf6328aa2cd903f0a7d4ae5a0fffcd3dfa1d44aa2b80990370e3225
Oct 30 22:16:57.507157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:57.507131    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:57.515255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:57.509944    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228"]
Oct 30 22:16:58.481072 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:58.480979    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228" event={"ID":"aa389a05-73a7-490b-8d2d-a8e34d82b0af","Type":"ContainerStarted","Data":"0a9012f3ee7a69fa2936a425036de23ab089c90d6eb964df79f466ed036bdd93"}
Oct 30 22:16:58.481538 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:58.481082    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228"
Oct 30 22:16:58.481538 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:58.481101    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228" event={"ID":"aa389a05-73a7-490b-8d2d-a8e34d82b0af","Type":"ContainerStarted","Data":"84698186aaf6328aa2cd903f0a7d4ae5a0fffcd3dfa1d44aa2b80990370e3225"}
Oct 30 22:16:58.538272 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:58.537815    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228" podStartSLOduration=2.045506541 podStartE2EDuration="2.537796742s" totalImagesPullingTime="492.290201ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:58.532496578 +0000 UTC m=+603.888862894" watchObservedRunningTime="2026-10-30 22:16:58.537796742 +0000 UTC m=+603.894163072"
Oct 30 22:16:59.343894 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:59.343856    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:59.435271 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:59.435213    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc"
Oct 30 22:16:59.462659 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:59.462635    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7"
Oct 30 22:16:59.541293 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:16:59.541236    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:00.437930 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:00.437899    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6"
Oct 30 22:17:00.959704 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:00.959666    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:02.767879 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:02.767841    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:04.500777 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:04.499787    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228"
Oct 30 22:17:05.029152 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:05.029078    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:06.296854 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:06.296808    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:06.565109 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:06.565031    4889 generic.go:356] "Generic (PLEG): container finished" podID="137329f9-9437-46ab-b030-ead038ccd542" containerID="b95598dfcf28df436f7a2d55ed03f23337e74d6cf084ac26b9ceca13f16a5dc0" exitCode=0
Oct 30 22:17:06.565344 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:06.565109    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt" event={"ID":"137329f9-9437-46ab-b030-ead038ccd542","Type":"ContainerDied","Data":"b95598dfcf28df436f7a2d55ed03f23337e74d6cf084ac26b9ceca13f16a5dc0"}
Oct 30 22:17:06.565837 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:06.565821    4889 scope.go:122] "RemoveContainer" containerID="b95598dfcf28df436f7a2d55ed03f23337e74d6cf084ac26b9ceca13f16a5dc0"
Oct 30 22:17:06.567218 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:06.567191    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:17:06.567330 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:06.567287    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:17:06.567392 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:06.567341    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:17:06.967308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:06.960428    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:07.135751 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:07.135720    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:07.320250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:07.314808    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:07.592284 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:07.592173    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt" event={"ID":"137329f9-9437-46ab-b030-ead038ccd542","Type":"ContainerStarted","Data":"c49ea9e2f2ae9fc1a105e6a0647ad8743c92428eea6417eeb109bb0a6837ba3c"}
Oct 30 22:17:07.690279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:07.689448    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:17:07.691359 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:07.691328    4889 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 22:17:07.691507 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:07.691394    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt" podUID="137329f9-9437-46ab-b030-ead038ccd542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 22:17:08.336102 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:08.336055    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:08.688104 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:08.688033    4889 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 22:17:08.688104 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:08.688087    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt" podUID="137329f9-9437-46ab-b030-ead038ccd542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 22:17:09.494475 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:09.494213    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:09.687984 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:09.687946    4889 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 22:17:09.688166 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:09.688013    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt" podUID="137329f9-9437-46ab-b030-ead038ccd542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 22:17:10.308184 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:17:10.308149    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:17:10.343113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:10.343036    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:10.351489 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:10.350372    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:17:10.384839 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:10.384389    4889 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:17:10.384839 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:10.384448    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg" podUID="7221740b-e003-4af1-86a2-396e32fff7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:10.384839 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:10.384753    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:17:10.384839 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:10.384786    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t" podUID="32e978e1-1585-4947-8975-fe20d4942bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:10.687965 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:10.687876    4889 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 22:17:10.687965 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:10.687939    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt" podUID="137329f9-9437-46ab-b030-ead038ccd542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 22:17:10.936004 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:10.935957    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:11.123332 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:11.123243    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:11.369161 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:11.369125    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:11.547304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:11.547266    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:11.688195 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:11.688163    4889 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 22:17:11.688587 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:11.688215    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt" podUID="137329f9-9437-46ab-b030-ead038ccd542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 22:17:11.979072 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:11.978992    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:12.208253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:12.208193    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:12.380515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:12.380455    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:12.688380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:12.688299    4889 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 22:17:12.688380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:12.688355    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt" podUID="137329f9-9437-46ab-b030-ead038ccd542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 22:17:13.638856 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:13.638827    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:13.650536 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:13.650508    4889 generic.go:356] "Generic (PLEG): container finished" podID="7221740b-e003-4af1-86a2-396e32fff7ca" containerID="d385f1fd17d203664f64a9a612f0007617ba893cef05ea957423d7e0b3cc2f95" exitCode=0
Oct 30 22:17:13.650686 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:13.650585    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg" event={"ID":"7221740b-e003-4af1-86a2-396e32fff7ca","Type":"ContainerDied","Data":"d385f1fd17d203664f64a9a612f0007617ba893cef05ea957423d7e0b3cc2f95"}
Oct 30 22:17:13.651060 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:13.651042    4889 scope.go:122] "RemoveContainer" containerID="d385f1fd17d203664f64a9a612f0007617ba893cef05ea957423d7e0b3cc2f95"
Oct 30 22:17:13.652281 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:13.652260    4889 generic.go:356] "Generic (PLEG): container finished" podID="32e978e1-1585-4947-8975-fe20d4942bc5" containerID="c7bcd4f15b101168d04b0b906613054c5076a4441a0a6202a6bdec339694c1f1" exitCode=0
Oct 30 22:17:13.652382 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:13.652322    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t" event={"ID":"32e978e1-1585-4947-8975-fe20d4942bc5","Type":"ContainerDied","Data":"c7bcd4f15b101168d04b0b906613054c5076a4441a0a6202a6bdec339694c1f1"}
Oct 30 22:17:13.652382 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:13.652353    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:17:13.652465 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:13.652416    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:17:13.652465 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:13.652455    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:17:13.652622 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:13.652609    4889 scope.go:122] "RemoveContainer" containerID="c7bcd4f15b101168d04b0b906613054c5076a4441a0a6202a6bdec339694c1f1"
Oct 30 22:17:13.653758 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:13.653734    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:17:13.653851 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:13.653791    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:17:13.653851 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:13.653837    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:17:13.692405 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:13.692381    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:17:13.693435 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:13.693417    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:17:14.668014 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:14.667983    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg" event={"ID":"7221740b-e003-4af1-86a2-396e32fff7ca","Type":"ContainerStarted","Data":"b9bef27d10252e835d86a70b9b33765d31305448b82624b38a8434991f107d55"}
Oct 30 22:17:14.670401 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:14.670376    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t" event={"ID":"32e978e1-1585-4947-8975-fe20d4942bc5","Type":"ContainerStarted","Data":"2bde103a7436ca298adbd1ab5c82638f9ac58bd4b7aee7b0a3fd1e41fe9cef14"}
Oct 30 22:17:15.144252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:15.140649    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:17:15.144252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:15.140787    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 22:17:15.144252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:15.140833    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t" podUID="32e978e1-1585-4947-8975-fe20d4942bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 22:17:15.383141 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:15.383093    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:17:15.383368 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:15.383311    4889 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 22:17:15.383368 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:15.383356    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg" podUID="7221740b-e003-4af1-86a2-396e32fff7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 22:17:15.544247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:15.536348    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:16.144119 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:16.143642    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 22:17:16.144119 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:16.143706    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t" podUID="32e978e1-1585-4947-8975-fe20d4942bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 22:17:16.382511 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:16.382479    4889 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 22:17:16.382684 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:16.382529    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg" podUID="7221740b-e003-4af1-86a2-396e32fff7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 22:17:16.653956 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:16.653870    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:16.837250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:16.834129    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:17.143250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:17.142622    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 22:17:17.143250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:17.142687    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t" podUID="32e978e1-1585-4947-8975-fe20d4942bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 22:17:17.302772 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:17.301368    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:17.382934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:17.382881    4889 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 22:17:17.383270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:17.382951    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg" podUID="7221740b-e003-4af1-86a2-396e32fff7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 22:17:17.491524 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:17.491439    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:17.673297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:17.673053    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:17.873421 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:17.873328    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:18.144310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:18.143613    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 22:17:18.144310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:18.143679    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t" podUID="32e978e1-1585-4947-8975-fe20d4942bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 22:17:18.383023 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:18.382977    4889 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 22:17:18.383462 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:18.383037    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg" podUID="7221740b-e003-4af1-86a2-396e32fff7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 22:17:19.141249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:19.140630    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 22:17:19.141249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:19.140701    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t" podUID="32e978e1-1585-4947-8975-fe20d4942bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 22:17:19.382884 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:19.382846    4889 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 22:17:19.383129 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:19.383108    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg" podUID="7221740b-e003-4af1-86a2-396e32fff7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 22:17:19.879266 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:19.879139    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:20.146551 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:20.146472    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:17:20.146930 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:20.146911    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:17:20.147690 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:20.147670    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:17:20.386871 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:20.386845    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:17:20.387326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:20.387089    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:17:20.387836 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:20.387819    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:17:21.373681 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:21.373638    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:21.771258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:21.771176    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:21.972984 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:21.972953    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:22.398922 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:22.398890    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:22.583380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:22.583349    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:23.666278 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:23.666242    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:23.855977 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:23.855938    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:25.445465 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:25.445429    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:27.282251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:27.282188    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:28.601054 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:28.601017    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:29.219765 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:29.219723    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:30.027683 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.027651    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"]
Oct 30 22:17:30.031513 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.031490    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:17:30.049870 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.049841    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"]
Oct 30 22:17:30.060064 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.060033    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0ad7c5\"/\"simulator-w4-0ad7c5-kserve-self-signed-certs\""
Oct 30 22:17:30.060064 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.060040    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0ad7c5\"/\"default-dockercfg-gn2fb\""
Oct 30 22:17:30.060327 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.060116    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0ad7c5\"/\"kube-root-ca.crt\""
Oct 30 22:17:30.060423 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.060048    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0ad7c5\"/\"openshift-service-ca.crt\""
Oct 30 22:17:30.136368 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.136339    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-model-cache\") pod \"simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp\" (UID: \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\") " pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:17:30.136526 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.136395    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-dshm\") pod \"simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp\" (UID: \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\") " pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:17:30.136526 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.136471    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-home\") pod \"simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp\" (UID: \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\") " pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:17:30.136526 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.136524    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-tls-certs\") pod \"simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp\" (UID: \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\") " pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:17:30.136678 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.136592    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzlmh\" (UniqueName: \"kubernetes.io/projected/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-kube-api-access-pzlmh\") pod \"simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp\" (UID: \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\") " pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:17:30.136678 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.136613    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-tmp-dir\") pod \"simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp\" (UID: \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\") " pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:17:30.155911 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.155881    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:30.237274 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.237205    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzlmh\" (UniqueName: \"kubernetes.io/projected/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-kube-api-access-pzlmh\") pod \"simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp\" (UID: \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\") " pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:17:30.237274 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.237281    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-tmp-dir\") pod \"simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp\" (UID: \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\") " pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:17:30.237556 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.237308    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-model-cache\") pod \"simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp\" (UID: \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\") " pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:17:30.237556 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.237370    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-dshm\") pod \"simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp\" (UID: \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\") " pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:17:30.237556 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.237389    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-home\") pod \"simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp\" (UID: \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\") " pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:17:30.237556 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.237410    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-tls-certs\") pod \"simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp\" (UID: \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\") " pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:17:30.237793 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.237770    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-tmp-dir\") pod \"simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp\" (UID: \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\") " pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:17:30.237854 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.237807    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-home\") pod \"simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp\" (UID: \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\") " pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:17:30.237854 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.237832    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-model-cache\") pod \"simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp\" (UID: \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\") " pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:17:30.239864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.239843    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-dshm\") pod \"simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp\" (UID: \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\") " pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:17:30.240148 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.240130    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-tls-certs\") pod \"simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp\" (UID: \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\") " pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:17:30.285303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.279774    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzlmh\" (UniqueName: \"kubernetes.io/projected/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-kube-api-access-pzlmh\") pod \"simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp\" (UID: \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\") " pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:17:30.344109 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.344075    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:17:30.643145 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.643115    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"]
Oct 30 22:17:30.643755 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:17:30.643725    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f4c1b10_0d9c_4bb7_a49a_df55737c6a05.slice/crio-6671f0e60d3606eab5f5a826041f23bf5261a01ec88a74c0e4ea5e7296e19a7f WatchSource:0}: Error finding container 6671f0e60d3606eab5f5a826041f23bf5261a01ec88a74c0e4ea5e7296e19a7f: Status 404 returned error can't find the container with id 6671f0e60d3606eab5f5a826041f23bf5261a01ec88a74c0e4ea5e7296e19a7f
Oct 30 22:17:30.781389 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.781357    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" event={"ID":"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05","Type":"ContainerStarted","Data":"9979ecfa6ebabfae18e7be2303f8813d1b85e720a9fc2de0b763b034069a322f"}
Oct 30 22:17:30.781584 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.781398    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" event={"ID":"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05","Type":"ContainerStarted","Data":"6671f0e60d3606eab5f5a826041f23bf5261a01ec88a74c0e4ea5e7296e19a7f"}
Oct 30 22:17:30.828344 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.828289    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podStartSLOduration=0.828273699 podStartE2EDuration="828.273699ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:30.825300122 +0000 UTC m=+636.181666434" watchObservedRunningTime="2026-10-30 22:17:30.828273699 +0000 UTC m=+636.184640011"
Oct 30 22:17:30.973644 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:30.973544    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:31.344748 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:31.344711    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:17:32.664036 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:32.664001    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:34.475674 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:34.475640    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:36.221682 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:36.221646    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:37.861028 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:37.860988    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:38.030194 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:38.030159    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:39.740464 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:39.740430    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:40.303024 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:17:40.302986    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:17:40.344725 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:40.344681    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:17:40.361162 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:40.361134    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:17:40.846098 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:40.846065    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:17:41.431255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:41.431198    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:43.253034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:43.253000    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:45.036529 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:45.036491    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:45.522974 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:45.522928    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:46.123524 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:46.123488    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:46.716126 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:46.716088    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:48.404586 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:48.403647    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:50.281880 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:50.281833    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:52.018773 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:52.018730    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:53.318845 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.318805    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"]
Oct 30 22:17:53.322171 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.322150    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:17:53.340105 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.340074    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"]
Oct 30 22:17:53.350037 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.350005    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-2b1e88\"/\"kube-root-ca.crt\""
Oct 30 22:17:53.350037 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.350034    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-2b1e88\"/\"premium-w1-2b1e88-kserve-self-signed-certs\""
Oct 30 22:17:53.350309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.350013    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-2b1e88\"/\"openshift-service-ca.crt\""
Oct 30 22:17:53.350309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.350138    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-2b1e88\"/\"default-dockercfg-pm7m5\""
Oct 30 22:17:53.416893 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.416855    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lc5g\" (UniqueName: \"kubernetes.io/projected/5d18c6ee-2f7b-4a85-9065-6c925004973d-kube-api-access-5lc5g\") pod \"premium-w1-2b1e88-kserve-5867fcd7-scd46\" (UID: \"5d18c6ee-2f7b-4a85-9065-6c925004973d\") " pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:17:53.417061 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.416914    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-tmp-dir\") pod \"premium-w1-2b1e88-kserve-5867fcd7-scd46\" (UID: \"5d18c6ee-2f7b-4a85-9065-6c925004973d\") " pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:17:53.417061 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.417008    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d18c6ee-2f7b-4a85-9065-6c925004973d-tls-certs\") pod \"premium-w1-2b1e88-kserve-5867fcd7-scd46\" (UID: \"5d18c6ee-2f7b-4a85-9065-6c925004973d\") " pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:17:53.417138 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.417113    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-model-cache\") pod \"premium-w1-2b1e88-kserve-5867fcd7-scd46\" (UID: \"5d18c6ee-2f7b-4a85-9065-6c925004973d\") " pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:17:53.417175 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.417154    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-home\") pod \"premium-w1-2b1e88-kserve-5867fcd7-scd46\" (UID: \"5d18c6ee-2f7b-4a85-9065-6c925004973d\") " pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:17:53.417175 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.417171    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-dshm\") pod \"premium-w1-2b1e88-kserve-5867fcd7-scd46\" (UID: \"5d18c6ee-2f7b-4a85-9065-6c925004973d\") " pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:17:53.440031 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.439998    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:53.563305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.561753    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d18c6ee-2f7b-4a85-9065-6c925004973d-tls-certs\") pod \"premium-w1-2b1e88-kserve-5867fcd7-scd46\" (UID: \"5d18c6ee-2f7b-4a85-9065-6c925004973d\") " pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:17:53.563305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.561856    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-model-cache\") pod \"premium-w1-2b1e88-kserve-5867fcd7-scd46\" (UID: \"5d18c6ee-2f7b-4a85-9065-6c925004973d\") " pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:17:53.563305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.561896    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-home\") pod \"premium-w1-2b1e88-kserve-5867fcd7-scd46\" (UID: \"5d18c6ee-2f7b-4a85-9065-6c925004973d\") " pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:17:53.563305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.561921    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-dshm\") pod \"premium-w1-2b1e88-kserve-5867fcd7-scd46\" (UID: \"5d18c6ee-2f7b-4a85-9065-6c925004973d\") " pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:17:53.563305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.561983    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lc5g\" (UniqueName: \"kubernetes.io/projected/5d18c6ee-2f7b-4a85-9065-6c925004973d-kube-api-access-5lc5g\") pod \"premium-w1-2b1e88-kserve-5867fcd7-scd46\" (UID: \"5d18c6ee-2f7b-4a85-9065-6c925004973d\") " pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:17:53.563305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.562059    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-tmp-dir\") pod \"premium-w1-2b1e88-kserve-5867fcd7-scd46\" (UID: \"5d18c6ee-2f7b-4a85-9065-6c925004973d\") " pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:17:53.563305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.562599    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-tmp-dir\") pod \"premium-w1-2b1e88-kserve-5867fcd7-scd46\" (UID: \"5d18c6ee-2f7b-4a85-9065-6c925004973d\") " pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:17:53.563305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.563162    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-home\") pod \"premium-w1-2b1e88-kserve-5867fcd7-scd46\" (UID: \"5d18c6ee-2f7b-4a85-9065-6c925004973d\") " pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:17:53.563865 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.563409    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-model-cache\") pod \"premium-w1-2b1e88-kserve-5867fcd7-scd46\" (UID: \"5d18c6ee-2f7b-4a85-9065-6c925004973d\") " pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:17:53.569734 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.569653    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d18c6ee-2f7b-4a85-9065-6c925004973d-tls-certs\") pod \"premium-w1-2b1e88-kserve-5867fcd7-scd46\" (UID: \"5d18c6ee-2f7b-4a85-9065-6c925004973d\") " pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:17:53.570400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.570373    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-dshm\") pod \"premium-w1-2b1e88-kserve-5867fcd7-scd46\" (UID: \"5d18c6ee-2f7b-4a85-9065-6c925004973d\") " pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:17:53.584312 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.584277    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lc5g\" (UniqueName: \"kubernetes.io/projected/5d18c6ee-2f7b-4a85-9065-6c925004973d-kube-api-access-5lc5g\") pod \"premium-w1-2b1e88-kserve-5867fcd7-scd46\" (UID: \"5d18c6ee-2f7b-4a85-9065-6c925004973d\") " pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:17:53.637159 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.637122    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:17:53.821806 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.821662    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:53.935130 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:53.935096    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"]
Oct 30 22:17:53.935441 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:17:53.935418    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d18c6ee_2f7b_4a85_9065_6c925004973d.slice/crio-1d0584ecfe1c12dfdddb183bd9c8f0bd436371690081b58e092dd7fd941dd30b WatchSource:0}: Error finding container 1d0584ecfe1c12dfdddb183bd9c8f0bd436371690081b58e092dd7fd941dd30b: Status 404 returned error can't find the container with id 1d0584ecfe1c12dfdddb183bd9c8f0bd436371690081b58e092dd7fd941dd30b
Oct 30 22:17:54.913130 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:54.913094    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" event={"ID":"5d18c6ee-2f7b-4a85-9065-6c925004973d","Type":"ContainerStarted","Data":"d67e46092f17279a11459e9f4ef2bfffd7e07d98886da23307f1410590683891"}
Oct 30 22:17:54.913496 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:54.913149    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" event={"ID":"5d18c6ee-2f7b-4a85-9065-6c925004973d","Type":"ContainerStarted","Data":"1d0584ecfe1c12dfdddb183bd9c8f0bd436371690081b58e092dd7fd941dd30b"}
Oct 30 22:17:54.952917 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:54.952869    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podStartSLOduration=1.952854876 podStartE2EDuration="1.952854876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:54.95163665 +0000 UTC m=+660.308002966" watchObservedRunningTime="2026-10-30 22:17:54.952854876 +0000 UTC m=+660.309221190"
Oct 30 22:17:55.620929 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:55.618985    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:55.637653 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:55.637620    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:17:57.416569 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:57.416528    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:59.217262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:17:59.213389    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:00.596765 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:00.596729    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:00.907159 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:00.907080    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:02.690983 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:02.690929    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:03.637795 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:03.637758    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:18:03.651065 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:03.651033    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:18:03.974962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:03.974861    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:18:04.083914 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.083885    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"]
Oct 30 22:18:04.087437 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.087411    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:18:04.104088 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.104055    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-09d0d5\"/\"openshift-service-ca.crt\""
Oct 30 22:18:04.104088 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.104059    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-09d0d5\"/\"default-dockercfg-69b2d\""
Oct 30 22:18:04.104354 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.104098    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-09d0d5\"/\"premium-w0-09d0d5-kserve-self-signed-certs\""
Oct 30 22:18:04.104516 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.104489    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-09d0d5\"/\"kube-root-ca.crt\""
Oct 30 22:18:04.106448 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.106427    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"]
Oct 30 22:18:04.177428 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.177393    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6af5c6ee-e62b-4095-924b-4d4440b91a26-tls-certs\") pod \"premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x\" (UID: \"6af5c6ee-e62b-4095-924b-4d4440b91a26\") " pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:18:04.177624 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.177453    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-tmp-dir\") pod \"premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x\" (UID: \"6af5c6ee-e62b-4095-924b-4d4440b91a26\") " pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:18:04.177624 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.177593    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfnpq\" (UniqueName: \"kubernetes.io/projected/6af5c6ee-e62b-4095-924b-4d4440b91a26-kube-api-access-jfnpq\") pod \"premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x\" (UID: \"6af5c6ee-e62b-4095-924b-4d4440b91a26\") " pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:18:04.177716 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.177653    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-dshm\") pod \"premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x\" (UID: \"6af5c6ee-e62b-4095-924b-4d4440b91a26\") " pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:18:04.177716 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.177698    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-model-cache\") pod \"premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x\" (UID: \"6af5c6ee-e62b-4095-924b-4d4440b91a26\") " pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:18:04.177792 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.177752    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-home\") pod \"premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x\" (UID: \"6af5c6ee-e62b-4095-924b-4d4440b91a26\") " pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:18:04.214852 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.214816    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:04.278715 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.278680    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-home\") pod \"premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x\" (UID: \"6af5c6ee-e62b-4095-924b-4d4440b91a26\") " pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:18:04.278900 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.278766    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6af5c6ee-e62b-4095-924b-4d4440b91a26-tls-certs\") pod \"premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x\" (UID: \"6af5c6ee-e62b-4095-924b-4d4440b91a26\") " pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:18:04.278900 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.278802    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-tmp-dir\") pod \"premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x\" (UID: \"6af5c6ee-e62b-4095-924b-4d4440b91a26\") " pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:18:04.278900 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.278856    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfnpq\" (UniqueName: \"kubernetes.io/projected/6af5c6ee-e62b-4095-924b-4d4440b91a26-kube-api-access-jfnpq\") pod \"premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x\" (UID: \"6af5c6ee-e62b-4095-924b-4d4440b91a26\") " pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:18:04.279047 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.278901    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-dshm\") pod \"premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x\" (UID: \"6af5c6ee-e62b-4095-924b-4d4440b91a26\") " pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:18:04.279047 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.278919    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-model-cache\") pod \"premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x\" (UID: \"6af5c6ee-e62b-4095-924b-4d4440b91a26\") " pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:18:04.279187 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.279163    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-home\") pod \"premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x\" (UID: \"6af5c6ee-e62b-4095-924b-4d4440b91a26\") " pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:18:04.279280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.279190    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-tmp-dir\") pod \"premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x\" (UID: \"6af5c6ee-e62b-4095-924b-4d4440b91a26\") " pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:18:04.279375 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.279358    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-model-cache\") pod \"premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x\" (UID: \"6af5c6ee-e62b-4095-924b-4d4440b91a26\") " pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:18:04.281340 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.281319    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-dshm\") pod \"premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x\" (UID: \"6af5c6ee-e62b-4095-924b-4d4440b91a26\") " pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:18:04.281549 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.281530    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6af5c6ee-e62b-4095-924b-4d4440b91a26-tls-certs\") pod \"premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x\" (UID: \"6af5c6ee-e62b-4095-924b-4d4440b91a26\") " pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:18:04.298356 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.298326    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfnpq\" (UniqueName: \"kubernetes.io/projected/6af5c6ee-e62b-4095-924b-4d4440b91a26-kube-api-access-jfnpq\") pod \"premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x\" (UID: \"6af5c6ee-e62b-4095-924b-4d4440b91a26\") " pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:18:04.395534 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.395496    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:04.401192 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.401164    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:18:04.708235 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.708190    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"]
Oct 30 22:18:04.708680 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:18:04.708658    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6af5c6ee_e62b_4095_924b_4d4440b91a26.slice/crio-0ae6bcf98c17623064d288d006d762dbba06c751202c1cf4f2c8a0f78919e936 WatchSource:0}: Error finding container 0ae6bcf98c17623064d288d006d762dbba06c751202c1cf4f2c8a0f78919e936: Status 404 returned error can't find the container with id 0ae6bcf98c17623064d288d006d762dbba06c751202c1cf4f2c8a0f78919e936
Oct 30 22:18:04.971917 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.971770    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" event={"ID":"6af5c6ee-e62b-4095-924b-4d4440b91a26","Type":"ContainerStarted","Data":"8e3568151d99459f8b720d04282c5af74325a9f235b6ca40dfd45f0fdbb66212"}
Oct 30 22:18:04.971917 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:04.971832    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" event={"ID":"6af5c6ee-e62b-4095-924b-4d4440b91a26","Type":"ContainerStarted","Data":"0ae6bcf98c17623064d288d006d762dbba06c751202c1cf4f2c8a0f78919e936"}
Oct 30 22:18:05.017707 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:05.017651    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podStartSLOduration=1.017636228 podStartE2EDuration="1.017636228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:05.015787982 +0000 UTC m=+670.372154294" watchObservedRunningTime="2026-10-30 22:18:05.017636228 +0000 UTC m=+670.374002543"
Oct 30 22:18:05.402338 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:05.402300    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:18:06.125344 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:06.125305    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:06.455438 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:06.455350    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:07.748889 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:07.748860    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:07.917369 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:07.917333    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:09.584091 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:09.584053    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:10.109178 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:10.109139    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:10.301615 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:18:10.301583    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:18:11.339688 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:11.339654    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:13.137093 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:13.137058    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:14.401594 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:14.401561    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:18:14.414628 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:14.414596    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:18:14.829101 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:14.829042    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:15.044197 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:15.044168    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:18:16.621006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:16.620974    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:18.336277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:18.336209    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:18.504151 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:18.504110    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:20.185472 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:20.185438    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:22.030270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:22.030238    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:23.722675 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:23.722641    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:25.421800 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:25.421764    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:27.178865 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:27.178821    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:29.135146 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:29.135115    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:30.849946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:30.849897    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:32.668699 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:32.668660    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:32.902078 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:32.902039    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:34.317313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:34.316199    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:36.010141 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:36.010103    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:37.019648 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.019607    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"]
Oct 30 22:18:37.024858 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.024834    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.051978 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.051949    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-09d0d5-openshift-default-dockercfg-x6h8g\""
Oct 30 22:18:37.052401 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.052374    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.052531 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.052432    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.052531 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.052460    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.052531 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.052496    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.052714 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.052614    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.052714 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.052696    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08eb9eea-09b6-4149-9c58-5a252ef3062a-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.052922 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.052896    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.052989 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.052937    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22gwf\" (UniqueName: \"kubernetes.io/projected/08eb9eea-09b6-4149-9c58-5a252ef3062a-kube-api-access-22gwf\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.052989 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.052966    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.054866 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.054840    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"]
Oct 30 22:18:37.153720 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.153685    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.153720 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.153734    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08eb9eea-09b6-4149-9c58-5a252ef3062a-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.154011 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.153780    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.154011 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.153797    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22gwf\" (UniqueName: \"kubernetes.io/projected/08eb9eea-09b6-4149-9c58-5a252ef3062a-kube-api-access-22gwf\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.154011 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.153814    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.154011 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.153860    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.154011 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.153898    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.154011 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.153922    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.154011 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.153951    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.154428 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.154280    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.154504 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.154476    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.154577 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.154555    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.154682 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.154658    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.155098 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.155070    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08eb9eea-09b6-4149-9c58-5a252ef3062a-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.156612 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.156584    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.156777 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.156762    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.173782 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.173758    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.173982 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.173965    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22gwf\" (UniqueName: \"kubernetes.io/projected/08eb9eea-09b6-4149-9c58-5a252ef3062a-kube-api-access-22gwf\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.261754 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.261714    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"]
Oct 30 22:18:37.266936 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.266911    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.282376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.282349    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-54114e-openshift-default-dockercfg-x4lxn\""
Oct 30 22:18:37.285300 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.285275    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"]
Oct 30 22:18:37.340096 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.340061    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:37.355697 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.355670    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzvj8\" (UniqueName: \"kubernetes.io/projected/794ada71-4cea-4845-90da-72e562678b17-kube-api-access-lzvj8\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.355819 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.355717    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/794ada71-4cea-4845-90da-72e562678b17-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.355819 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.355738    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.355948 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.355821    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.355948 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.355931    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/794ada71-4cea-4845-90da-72e562678b17-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.356063 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.355987    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/794ada71-4cea-4845-90da-72e562678b17-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.356127 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.356108    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.356183 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.356170    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.356310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.356292    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.457861 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.457818    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.458051 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.457888    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.458051 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.457942    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.458051 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.457985    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzvj8\" (UniqueName: \"kubernetes.io/projected/794ada71-4cea-4845-90da-72e562678b17-kube-api-access-lzvj8\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.458051 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.458032    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/794ada71-4cea-4845-90da-72e562678b17-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.458280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.458059    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.458280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.458087    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.458280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.458119    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/794ada71-4cea-4845-90da-72e562678b17-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.458280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.458165    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/794ada71-4cea-4845-90da-72e562678b17-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.458996 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.458964    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.459130 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.459013    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.459215 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.459193    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.459382 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.459364    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.459616 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.459593    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/794ada71-4cea-4845-90da-72e562678b17-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.461060 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.461041    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.461320 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.461302    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/794ada71-4cea-4845-90da-72e562678b17-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.478505 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.478477    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzvj8\" (UniqueName: \"kubernetes.io/projected/794ada71-4cea-4845-90da-72e562678b17-kube-api-access-lzvj8\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.478620 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.478602    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/794ada71-4cea-4845-90da-72e562678b17-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.581862 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.581776    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:37.656377 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.656347    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"]
Oct 30 22:18:37.656550 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:18:37.656529    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08eb9eea_09b6_4149_9c58_5a252ef3062a.slice/crio-bb103cc1c89b7d16adbe28098777ba8cd40ac23f366093ab1e79b08a3cc5ff4c WatchSource:0}: Error finding container bb103cc1c89b7d16adbe28098777ba8cd40ac23f366093ab1e79b08a3cc5ff4c: Status 404 returned error can't find the container with id bb103cc1c89b7d16adbe28098777ba8cd40ac23f366093ab1e79b08a3cc5ff4c
Oct 30 22:18:37.659487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.659449    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:18:37.659575 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.659532    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:18:37.659615 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.659570    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:18:37.717857 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.717825    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:37.886476 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:18:37.886447    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794ada71_4cea_4845_90da_72e562678b17.slice/crio-1814d6a75cec6da2b206cd8834f7e79016af08a92edb6160615291e348f8c2ec WatchSource:0}: Error finding container 1814d6a75cec6da2b206cd8834f7e79016af08a92edb6160615291e348f8c2ec: Status 404 returned error can't find the container with id 1814d6a75cec6da2b206cd8834f7e79016af08a92edb6160615291e348f8c2ec
Oct 30 22:18:37.886649 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.886551    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"]
Oct 30 22:18:37.888895 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.888867    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:18:37.888971 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.888927    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:18:37.888971 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:37.888964    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:18:38.187004 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:38.186914    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8" event={"ID":"08eb9eea-09b6-4149-9c58-5a252ef3062a","Type":"ContainerStarted","Data":"802fd5d0b6f4a8a2dd4205d5e826ce855238318dc56569db2cfac5f641f8d94d"}
Oct 30 22:18:38.187004 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:38.186973    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8" event={"ID":"08eb9eea-09b6-4149-9c58-5a252ef3062a","Type":"ContainerStarted","Data":"bb103cc1c89b7d16adbe28098777ba8cd40ac23f366093ab1e79b08a3cc5ff4c"}
Oct 30 22:18:38.189093 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:38.189063    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c" event={"ID":"794ada71-4cea-4845-90da-72e562678b17","Type":"ContainerStarted","Data":"4db41ccaaf73d9ab578807990e02b911c3efd9da34d7e6310a5b4611af8023fa"}
Oct 30 22:18:38.189254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:38.189114    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c" event={"ID":"794ada71-4cea-4845-90da-72e562678b17","Type":"ContainerStarted","Data":"1814d6a75cec6da2b206cd8834f7e79016af08a92edb6160615291e348f8c2ec"}
Oct 30 22:18:38.243251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:38.235450    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8" podStartSLOduration=2.235430643 podStartE2EDuration="2.235430643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:38.234740216 +0000 UTC m=+703.591106532" watchObservedRunningTime="2026-10-30 22:18:38.235430643 +0000 UTC m=+703.591796959"
Oct 30 22:18:38.281752 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:38.281699    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c" podStartSLOduration=1.281680592 podStartE2EDuration="1.281680592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:38.280845585 +0000 UTC m=+703.637211899" watchObservedRunningTime="2026-10-30 22:18:38.281680592 +0000 UTC m=+703.638046907"
Oct 30 22:18:38.341441 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:38.341404    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:38.343010 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:38.342978    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 22:18:38.343152 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:38.343043    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8" podUID="08eb9eea-09b6-4149-9c58-5a252ef3062a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 22:18:38.582866 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:38.582836    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:38.584587 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:38.584562    4889 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 22:18:38.584727 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:38.584620    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c" podUID="794ada71-4cea-4845-90da-72e562678b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 22:18:39.341250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:39.340679    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 22:18:39.341250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:39.340753    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8" podUID="08eb9eea-09b6-4149-9c58-5a252ef3062a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 22:18:39.590249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:39.587162    4889 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 22:18:39.590249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:39.587248    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c" podUID="794ada71-4cea-4845-90da-72e562678b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 22:18:39.934253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:39.933218    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:40.301128 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:18:40.301087    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:18:40.349251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:40.340742    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 22:18:40.349251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:40.340822    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8" podUID="08eb9eea-09b6-4149-9c58-5a252ef3062a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 22:18:40.583250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:40.583160    4889 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 22:18:40.583402 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:40.583216    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c" podUID="794ada71-4cea-4845-90da-72e562678b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 22:18:41.351257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:41.350620    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 22:18:41.351257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:41.350681    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8" podUID="08eb9eea-09b6-4149-9c58-5a252ef3062a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 22:18:41.582296 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:41.582260    4889 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 22:18:41.582474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:41.582312    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c" podUID="794ada71-4cea-4845-90da-72e562678b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 22:18:42.098289 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:42.098163    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:42.344251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:42.342650    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 22:18:42.344251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:42.342730    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8" podUID="08eb9eea-09b6-4149-9c58-5a252ef3062a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 22:18:42.582965 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:42.582929    4889 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 22:18:42.583489 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:42.583027    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c" podUID="794ada71-4cea-4845-90da-72e562678b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 22:18:43.347726 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:43.347691    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 22:18:43.348012 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:43.347985    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8" podUID="08eb9eea-09b6-4149-9c58-5a252ef3062a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 22:18:43.582502 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:43.582472    4889 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 22:18:43.582667 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:43.582529    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c" podUID="794ada71-4cea-4845-90da-72e562678b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 22:18:44.273247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:44.273182    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:44.341196 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:44.341159    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 22:18:44.341434 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:44.341283    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8" podUID="08eb9eea-09b6-4149-9c58-5a252ef3062a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 22:18:44.583283 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:44.583176    4889 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 22:18:44.583510 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:44.583291    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c" podUID="794ada71-4cea-4845-90da-72e562678b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 22:18:45.652201 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:45.652160    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:45.724688 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:45.724655    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:45.725185 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:45.725164    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:45.726492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:45.726470    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:18:45.768004 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:45.767975    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:45.768283 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:45.768265    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:45.769018 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:45.769002    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:18:46.214352 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:46.213895    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:48.025159 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:48.025125    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:49.638580 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:49.638546    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:51.327465 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:51.327429    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:53.026609 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:53.026575    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:54.808369 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:54.808337    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:56.886606 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:56.886563    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:58.268633 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:18:58.268599    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:00.054416 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:00.054386    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:01.846821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:01.846776    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:03.536509 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:03.536466    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:05.230492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:05.230457    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:06.937168 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:06.937134    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:07.644063 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.644030    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"]
Oct 30 22:19:07.649397 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.649377    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.667882 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.667844    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-012ba0-openshift-default-dockercfg-gk2x2\""
Oct 30 22:19:07.682778 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.682749    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"]
Oct 30 22:19:07.737546 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.737515    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.737726 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.737569    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.737726 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.737591    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d441e1d3-9571-4513-8404-2d6569566ba9-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.737818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.737749    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.737818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.737792    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.737818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.737817    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d441e1d3-9571-4513-8404-2d6569566ba9-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.737951 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.737882    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.737951 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.737923    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d441e1d3-9571-4513-8404-2d6569566ba9-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.738049 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.738038    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jmxg\" (UniqueName: \"kubernetes.io/projected/d441e1d3-9571-4513-8404-2d6569566ba9-kube-api-access-8jmxg\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.839297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.839264    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.839469 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.839313    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.839469 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.839357    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d441e1d3-9571-4513-8404-2d6569566ba9-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.839469 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.839385    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.839469 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.839409    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d441e1d3-9571-4513-8404-2d6569566ba9-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.839647 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.839468    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jmxg\" (UniqueName: \"kubernetes.io/projected/d441e1d3-9571-4513-8404-2d6569566ba9-kube-api-access-8jmxg\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.839647 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.839561    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.839647 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.839635    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.839766 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.839661    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d441e1d3-9571-4513-8404-2d6569566ba9-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.839816 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.839759    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.839816 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.839774    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.840193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.840171    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.840637 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.840611    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d441e1d3-9571-4513-8404-2d6569566ba9-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.840995 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.840972    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.842148 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.842129    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.842524 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.842503    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d441e1d3-9571-4513-8404-2d6569566ba9-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.859587 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.859568    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d441e1d3-9571-4513-8404-2d6569566ba9-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.859724 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.859704    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jmxg\" (UniqueName: \"kubernetes.io/projected/d441e1d3-9571-4513-8404-2d6569566ba9-kube-api-access-8jmxg\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:07.963099 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:07.963022    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:08.286291 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:08.286263    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"]
Oct 30 22:19:08.287082 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:19:08.287060    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd441e1d3_9571_4513_8404_2d6569566ba9.slice/crio-e5b8a4c4b494e28de6fadf7f57dce0d25af6b8364e25c4efc7d03a7be3f6aa8f WatchSource:0}: Error finding container e5b8a4c4b494e28de6fadf7f57dce0d25af6b8364e25c4efc7d03a7be3f6aa8f: Status 404 returned error can't find the container with id e5b8a4c4b494e28de6fadf7f57dce0d25af6b8364e25c4efc7d03a7be3f6aa8f
Oct 30 22:19:08.289599 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:08.289565    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:19:08.289701 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:08.289648    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:19:08.289701 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:08.289694    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:19:08.455886 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:08.455849    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw" event={"ID":"d441e1d3-9571-4513-8404-2d6569566ba9","Type":"ContainerStarted","Data":"052197b16869f136f5ff20a6390908b88fbaf7f035d1d121f3a066b2a867b073"}
Oct 30 22:19:08.456072 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:08.455911    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw" event={"ID":"d441e1d3-9571-4513-8404-2d6569566ba9","Type":"ContainerStarted","Data":"e5b8a4c4b494e28de6fadf7f57dce0d25af6b8364e25c4efc7d03a7be3f6aa8f"}
Oct 30 22:19:08.501169 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:08.501117    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw" podStartSLOduration=1.501102765 podStartE2EDuration="1.501102765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:08.498769377 +0000 UTC m=+733.855135693" watchObservedRunningTime="2026-10-30 22:19:08.501102765 +0000 UTC m=+733.857469082"
Oct 30 22:19:08.746625 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:08.746591    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:08.963582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:08.963552    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:09.965322 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:09.965286    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:19:09.965778 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:09.965342    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw" podUID="d441e1d3-9571-4513-8404-2d6569566ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 22:19:10.302509 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:19:10.302469    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:19:10.742517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:10.742216    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:10.963435 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:10.963399    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:19:10.963629 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:10.963459    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw" podUID="d441e1d3-9571-4513-8404-2d6569566ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 22:19:11.964340 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:11.964306    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:19:11.964787 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:11.964363    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw" podUID="d441e1d3-9571-4513-8404-2d6569566ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 22:19:12.560941 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:12.557312    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:12.963740 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:12.963602    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:19:12.963740 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:12.963699    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw" podUID="d441e1d3-9571-4513-8404-2d6569566ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 22:19:13.964165 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:13.964130    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:19:13.964892 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:13.964854    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw" podUID="d441e1d3-9571-4513-8404-2d6569566ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 22:19:14.372276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:14.372201    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:15.063211 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:15.063176    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:15.064175 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:15.064139    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:15.065582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:15.065561    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:19:16.039248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:16.031237    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:17.732604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:17.732566    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:19.691919 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:19.691881    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:21.434376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:21.433864    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:23.132130 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:23.132092    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:24.850392 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:24.849164    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:27.345192 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:27.345155    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:30.014283 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:30.014247    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:31.739677 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:31.739640    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:33.438292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:33.437843    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:35.208707 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:35.208623    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:36.480784 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.480741    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"]
Oct 30 22:19:36.488292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.488267    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:36.525705 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.525669    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"]
Oct 30 22:19:36.650871 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.650825    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8c8cz\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:36.674692 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.674635    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8c8cz\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:36.675127 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.675101    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8c8cz\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:36.675449 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.675429    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8c8cz\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:36.675677 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.675659    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8c8cz\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:36.685101 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.685030    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8c8cz\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:36.685347 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.685328    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8c8cz\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:36.686356 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.686335    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmdvw\" (UniqueName: \"kubernetes.io/projected/9fca6fa2-01cd-4f49-96c3-60da13e5c889-kube-api-access-dmdvw\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8c8cz\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:36.686568 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.686554    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8c8cz\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:36.796279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.789949    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8c8cz\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:36.796279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.790025    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8c8cz\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:36.796279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.790084    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8c8cz\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:36.796279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.790154    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8c8cz\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:36.796279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.790192    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8c8cz\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:36.796279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.790828    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8c8cz\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:36.796279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.796274    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8c8cz\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:36.796856 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.796317    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8c8cz\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:36.796856 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.796372    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmdvw\" (UniqueName: \"kubernetes.io/projected/9fca6fa2-01cd-4f49-96c3-60da13e5c889-kube-api-access-dmdvw\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8c8cz\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:36.796856 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.796406    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8c8cz\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:36.796856 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.796841    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8c8cz\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:36.800261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.797527    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8c8cz\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:36.835274 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.822583    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8c8cz\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:36.835274 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.822859    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8c8cz\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:36.874424 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.874296    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8c8cz\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:36.937492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.922030    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8c8cz\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:36.937492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.934009    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8c8cz\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:36.963250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:36.950915    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmdvw\" (UniqueName: \"kubernetes.io/projected/9fca6fa2-01cd-4f49-96c3-60da13e5c889-kube-api-access-dmdvw\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8c8cz\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:37.188979 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:37.188890    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:37.728376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:37.722136    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"]
Oct 30 22:19:37.735992 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:37.735938    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:19:37.736292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:37.736208    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:19:37.736457 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:37.736437    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:19:37.796457 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:37.796427    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz" event={"ID":"9fca6fa2-01cd-4f49-96c3-60da13e5c889","Type":"ContainerStarted","Data":"6fbfeb461df3144d86128b40ae1f86cc46d4cd355cda6f2ff0e0148f9d4dd9f8"}
Oct 30 22:19:38.389249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:38.387289    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:38.803913 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:38.803883    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz" event={"ID":"9fca6fa2-01cd-4f49-96c3-60da13e5c889","Type":"ContainerStarted","Data":"73c14c5ce12b23c158a3d4542fce534c4b13e1577f8d541bdffa24311c0883b6"}
Oct 30 22:19:38.866133 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:38.866070    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz" podStartSLOduration=2.866049769 podStartE2EDuration="2.866049769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:38.865720411 +0000 UTC m=+764.222086737" watchObservedRunningTime="2026-10-30 22:19:38.866049769 +0000 UTC m=+764.222416087"
Oct 30 22:19:39.061356 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:39.061043    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:39.207813 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:39.207780    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:39.232246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:39.230408    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8c8cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 22:19:39.232246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:39.230512    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz" podUID="9fca6fa2-01cd-4f49-96c3-60da13e5c889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 22:19:40.190615 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:40.190587    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8c8cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 22:19:40.191108 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:40.190638    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz" podUID="9fca6fa2-01cd-4f49-96c3-60da13e5c889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 22:19:40.301755 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:19:40.301719    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:19:40.444932 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:40.444850    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:41.190489 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:41.190461    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8c8cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 22:19:41.190667 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:41.190509    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz" podUID="9fca6fa2-01cd-4f49-96c3-60da13e5c889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 22:19:42.190211 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:42.190176    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8c8cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 22:19:42.190382 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:42.190269    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz" podUID="9fca6fa2-01cd-4f49-96c3-60da13e5c889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 22:19:42.284010 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:42.283965    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:43.190218 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:43.190181    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8c8cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 22:19:43.190393 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:43.190242    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz" podUID="9fca6fa2-01cd-4f49-96c3-60da13e5c889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 22:19:44.071870 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:44.071828    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:44.190811 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:44.190776    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8c8cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 22:19:44.190988 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:44.190858    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz" podUID="9fca6fa2-01cd-4f49-96c3-60da13e5c889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 22:19:45.190482 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:45.190448    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8c8cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 22:19:45.190927 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:45.190513    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz" podUID="9fca6fa2-01cd-4f49-96c3-60da13e5c889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 22:19:45.752248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:45.752178    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:45.991960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:45.991926    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:46.981916 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:46.981886    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:46.982438 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:46.982294    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:46.983426 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:46.983405    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:19:47.536973 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:47.536922    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:49.240807 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:49.240771    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:51.000137 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:51.000102    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:53.141433 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:53.141398    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:54.920631 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:54.920601    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:56.706166 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:56.706134    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:58.436463 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:58.436427    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:59.074309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:19:59.074281    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:00.277104 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:00.276093    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:00.748367 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:00.748336    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:02.055173 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:02.055137    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:02.293505 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:02.293475    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"]
Oct 30 22:20:02.302922 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:02.299138    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:20:02.314522 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:02.314494    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"]
Oct 30 22:20:02.325158 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:02.325126    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0ad7c5\"/\"distinct-w4-0ad7c5-kserve-self-signed-certs\""
Oct 30 22:20:02.354936 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:02.354906    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42e72bf1-014e-446f-902a-e3a8ba06d34b-tls-certs\") pod \"distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f\" (UID: \"42e72bf1-014e-446f-902a-e3a8ba06d34b\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:20:02.355106 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:02.354967    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-tmp-dir\") pod \"distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f\" (UID: \"42e72bf1-014e-446f-902a-e3a8ba06d34b\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:20:02.355106 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:02.355069    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-home\") pod \"distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f\" (UID: \"42e72bf1-014e-446f-902a-e3a8ba06d34b\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:20:02.355186 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:02.355162    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-model-cache\") pod \"distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f\" (UID: \"42e72bf1-014e-446f-902a-e3a8ba06d34b\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:20:02.355252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:02.355192    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f4xs\" (UniqueName: \"kubernetes.io/projected/42e72bf1-014e-446f-902a-e3a8ba06d34b-kube-api-access-5f4xs\") pod \"distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f\" (UID: \"42e72bf1-014e-446f-902a-e3a8ba06d34b\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:20:02.355292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:02.355213    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-dshm\") pod \"distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f\" (UID: \"42e72bf1-014e-446f-902a-e3a8ba06d34b\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:20:02.419414 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:02.419379    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:02.456844 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:02.456803    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-home\") pod \"distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f\" (UID: \"42e72bf1-014e-446f-902a-e3a8ba06d34b\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:20:02.457036 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:02.456896    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-model-cache\") pod \"distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f\" (UID: \"42e72bf1-014e-446f-902a-e3a8ba06d34b\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:20:02.457036 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:02.456937    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f4xs\" (UniqueName: \"kubernetes.io/projected/42e72bf1-014e-446f-902a-e3a8ba06d34b-kube-api-access-5f4xs\") pod \"distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f\" (UID: \"42e72bf1-014e-446f-902a-e3a8ba06d34b\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:20:02.457036 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:02.456965    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-dshm\") pod \"distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f\" (UID: \"42e72bf1-014e-446f-902a-e3a8ba06d34b\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:20:02.457199 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:02.457076    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42e72bf1-014e-446f-902a-e3a8ba06d34b-tls-certs\") pod \"distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f\" (UID: \"42e72bf1-014e-446f-902a-e3a8ba06d34b\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:20:02.457199 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:02.457157    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-tmp-dir\") pod \"distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f\" (UID: \"42e72bf1-014e-446f-902a-e3a8ba06d34b\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:20:02.457361 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:02.457208    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-home\") pod \"distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f\" (UID: \"42e72bf1-014e-446f-902a-e3a8ba06d34b\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:20:02.457361 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:02.457338    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-model-cache\") pod \"distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f\" (UID: \"42e72bf1-014e-446f-902a-e3a8ba06d34b\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:20:02.457523 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:02.457504    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-tmp-dir\") pod \"distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f\" (UID: \"42e72bf1-014e-446f-902a-e3a8ba06d34b\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:20:02.459378 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:02.459359    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-dshm\") pod \"distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f\" (UID: \"42e72bf1-014e-446f-902a-e3a8ba06d34b\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:20:02.459764 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:02.459744    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42e72bf1-014e-446f-902a-e3a8ba06d34b-tls-certs\") pod \"distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f\" (UID: \"42e72bf1-014e-446f-902a-e3a8ba06d34b\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:20:02.476864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:02.476838    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f4xs\" (UniqueName: \"kubernetes.io/projected/42e72bf1-014e-446f-902a-e3a8ba06d34b-kube-api-access-5f4xs\") pod \"distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f\" (UID: \"42e72bf1-014e-446f-902a-e3a8ba06d34b\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:20:02.615920 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:02.615834    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:20:02.916470 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:02.916442    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"]
Oct 30 22:20:02.916839 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:20:02.916819    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42e72bf1_014e_446f_902a_e3a8ba06d34b.slice/crio-2d816b85ebe81c7aaf82eb8c1d97a771aa557bb2cbc6fdb8319fa5ffe0c1f7fd WatchSource:0}: Error finding container 2d816b85ebe81c7aaf82eb8c1d97a771aa557bb2cbc6fdb8319fa5ffe0c1f7fd: Status 404 returned error can't find the container with id 2d816b85ebe81c7aaf82eb8c1d97a771aa557bb2cbc6fdb8319fa5ffe0c1f7fd
Oct 30 22:20:03.004990 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:03.004951    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" event={"ID":"42e72bf1-014e-446f-902a-e3a8ba06d34b","Type":"ContainerStarted","Data":"652b969d08f26d003c44a6d38d3b06b46778fd8d2cb3c79249a8c75f00cc43ab"}
Oct 30 22:20:03.004990 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:03.005004    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" event={"ID":"42e72bf1-014e-446f-902a-e3a8ba06d34b","Type":"ContainerStarted","Data":"2d816b85ebe81c7aaf82eb8c1d97a771aa557bb2cbc6fdb8319fa5ffe0c1f7fd"}
Oct 30 22:20:03.045024 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:03.044962    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podStartSLOduration=1.04494795 podStartE2EDuration="1.04494795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:03.044751327 +0000 UTC m=+788.401117642" watchObservedRunningTime="2026-10-30 22:20:03.04494795 +0000 UTC m=+788.401314264"
Oct 30 22:20:03.617251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:03.616335    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:20:04.129249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:04.125254    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:05.928597 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:05.927365    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:07.415301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.415270    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"]
Oct 30 22:20:07.419258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.419238    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.453394 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.453359    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.453394 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.453403    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.453670 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.453574    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.453670 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.453614    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.453799 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.453699    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgn2m\" (UniqueName: \"kubernetes.io/projected/9759df3a-c1a8-40e6-a087-dfd96b243c12-kube-api-access-kgn2m\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.453799 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.453732    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.453799 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.453749    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.453799 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.453792    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.454078 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.453812    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9759df3a-c1a8-40e6-a087-dfd96b243c12-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.454078 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.453841    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"]
Oct 30 22:20:07.554644 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.554610    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.554828 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.554665    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.554828 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.554743    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.554828 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.554778    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.554828 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.554807    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgn2m\" (UniqueName: \"kubernetes.io/projected/9759df3a-c1a8-40e6-a087-dfd96b243c12-kube-api-access-kgn2m\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.555070 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.554838    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.555070 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.554863    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.555070 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.554923    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.555070 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.554948    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9759df3a-c1a8-40e6-a087-dfd96b243c12-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.555708 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.555687    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.555821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.555741    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9759df3a-c1a8-40e6-a087-dfd96b243c12-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.555821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.555741    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.555941 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.555884    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.555941 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.555903    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.557956 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.557938    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.558306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.558282    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.574483 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.574460    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.575277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.575255    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgn2m\" (UniqueName: \"kubernetes.io/projected/9759df3a-c1a8-40e6-a087-dfd96b243c12-kube-api-access-kgn2m\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.747000 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.745917    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:07.765409 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:07.765111    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:08.075976 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:20:08.075950    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9759df3a_c1a8_40e6_a087_dfd96b243c12.slice/crio-0f54fef16b4955831ca35a23293cf17655b512ba1cd7d4174ec9f9d5ade433ad WatchSource:0}: Error finding container 0f54fef16b4955831ca35a23293cf17655b512ba1cd7d4174ec9f9d5ade433ad: Status 404 returned error can't find the container with id 0f54fef16b4955831ca35a23293cf17655b512ba1cd7d4174ec9f9d5ade433ad
Oct 30 22:20:08.076094 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.076025    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"]
Oct 30 22:20:08.078664 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.078630    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:20:08.078773 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.078697    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:20:08.078773 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.078726    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:20:08.123549 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.123524    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"]
Oct 30 22:20:08.129483 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.129460    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.151631 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.151606    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"]
Oct 30 22:20:08.159621 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.159592    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-421583-openshift-default-dockercfg-fp7vh\""
Oct 30 22:20:08.170049 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.170013    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-istio-envoy\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.170196 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.170167    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b69c09df-8ced-418f-92f4-6c9ead312931-istio-podinfo\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.170297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.170255    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-workload-certs\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.170359 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.170300    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b69c09df-8ced-418f-92f4-6c9ead312931-istiod-ca-cert\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.170482 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.170449    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-workload-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.170554 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.170506    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq5rr\" (UniqueName: \"kubernetes.io/projected/b69c09df-8ced-418f-92f4-6c9ead312931-kube-api-access-pq5rr\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.170554 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.170539    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b69c09df-8ced-418f-92f4-6c9ead312931-istio-token\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.170690 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.170673    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-istio-data\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.170929 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.170907    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-credential-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.272405 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.272371    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-istio-envoy\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.272595 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.272425    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b69c09df-8ced-418f-92f4-6c9ead312931-istio-podinfo\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.272595 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.272460    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-workload-certs\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.272595 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.272498    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b69c09df-8ced-418f-92f4-6c9ead312931-istiod-ca-cert\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.272595 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.272555    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-workload-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.272595 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.272582    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq5rr\" (UniqueName: \"kubernetes.io/projected/b69c09df-8ced-418f-92f4-6c9ead312931-kube-api-access-pq5rr\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.272873 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.272678    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b69c09df-8ced-418f-92f4-6c9ead312931-istio-token\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.272928 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.272889    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-istio-data\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.272988 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.272941    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-credential-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.272988 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.272972    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-workload-certs\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.273172 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.273148    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-istio-data\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.273429 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.273405    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-credential-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.273528 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.273453    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-workload-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.273879 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.273837    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b69c09df-8ced-418f-92f4-6c9ead312931-istiod-ca-cert\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.274999 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.274973    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-istio-envoy\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.275645 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.275629    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b69c09df-8ced-418f-92f4-6c9ead312931-istio-podinfo\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.292374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.292351    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b69c09df-8ced-418f-92f4-6c9ead312931-istio-token\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.292498 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.292393    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq5rr\" (UniqueName: \"kubernetes.io/projected/b69c09df-8ced-418f-92f4-6c9ead312931-kube-api-access-pq5rr\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.456638 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.456559    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:08.791582 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:20:08.791555    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69c09df_8ced_418f_92f4_6c9ead312931.slice/crio-5c5b97c8ddf0022e0b7032a64640da32511064e98b96caa515e9ae5488d25440 WatchSource:0}: Error finding container 5c5b97c8ddf0022e0b7032a64640da32511064e98b96caa515e9ae5488d25440: Status 404 returned error can't find the container with id 5c5b97c8ddf0022e0b7032a64640da32511064e98b96caa515e9ae5488d25440
Oct 30 22:20:08.792689 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.792666    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"]
Oct 30 22:20:08.794451 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.794424    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:20:08.794537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.794481    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:20:08.794537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:08.794529    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:20:09.088137 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:09.088044    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79" event={"ID":"9759df3a-c1a8-40e6-a087-dfd96b243c12","Type":"ContainerStarted","Data":"0bb85fba1e2d8a07e00919e24b53f97a98f42dce95c5d98d0962914e15e7404e"}
Oct 30 22:20:09.088137 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:09.088108    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79" event={"ID":"9759df3a-c1a8-40e6-a087-dfd96b243c12","Type":"ContainerStarted","Data":"0f54fef16b4955831ca35a23293cf17655b512ba1cd7d4174ec9f9d5ade433ad"}
Oct 30 22:20:09.090571 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:09.090543    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp" event={"ID":"b69c09df-8ced-418f-92f4-6c9ead312931","Type":"ContainerStarted","Data":"67df2fd3ffc77e25f94f95889dcc7ef39e0488f2859f3f0889073ff02dbd0fd2"}
Oct 30 22:20:09.090750 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:09.090594    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp" event={"ID":"b69c09df-8ced-418f-92f4-6c9ead312931","Type":"ContainerStarted","Data":"5c5b97c8ddf0022e0b7032a64640da32511064e98b96caa515e9ae5488d25440"}
Oct 30 22:20:09.185192 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:09.185131    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79" podStartSLOduration=2.185110436 podStartE2EDuration="2.185110436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:09.141602193 +0000 UTC m=+794.497968509" watchObservedRunningTime="2026-10-30 22:20:09.185110436 +0000 UTC m=+794.541476755"
Oct 30 22:20:09.186171 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:09.186129    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp" podStartSLOduration=1.186112048 podStartE2EDuration="1.186112048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:09.182682924 +0000 UTC m=+794.539049250" watchObservedRunningTime="2026-10-30 22:20:09.186112048 +0000 UTC m=+794.542478363"
Oct 30 22:20:09.457728 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:09.457629    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:09.461404 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:09.461365    4889 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:20:09.461592 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:09.461432    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp" podUID="b69c09df-8ced-418f-92f4-6c9ead312931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 22:20:09.602258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:09.595274    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:09.759121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:09.753913    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:09.759121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:09.758623    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 22:20:09.759121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:09.758694    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79" podUID="9759df3a-c1a8-40e6-a087-dfd96b243c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 22:20:10.303069 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:20:10.303022    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:20:10.460249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:10.459940    4889 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:20:10.460249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:10.460000    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp" podUID="b69c09df-8ced-418f-92f4-6c9ead312931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 22:20:10.753096 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:10.752618    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 22:20:10.753096 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:10.752676    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79" podUID="9759df3a-c1a8-40e6-a087-dfd96b243c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 22:20:11.457700 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:11.457657    4889 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:20:11.458020 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:11.457993    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp" podUID="b69c09df-8ced-418f-92f4-6c9ead312931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 22:20:11.576573 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:11.576506    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:11.757259 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:11.755667    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 22:20:11.757259 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:11.755732    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79" podUID="9759df3a-c1a8-40e6-a087-dfd96b243c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 22:20:12.459250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:12.458733    4889 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:20:12.459250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:12.458794    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp" podUID="b69c09df-8ced-418f-92f4-6c9ead312931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 22:20:12.649251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:12.648300    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:20:12.668206 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:12.668176    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:20:12.751248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:12.750048    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 22:20:12.751248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:12.750104    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79" podUID="9759df3a-c1a8-40e6-a087-dfd96b243c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 22:20:13.175275 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:13.175186    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:20:13.460307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:13.459672    4889 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:20:13.460307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:13.459733    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp" podUID="b69c09df-8ced-418f-92f4-6c9ead312931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 22:20:13.768301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:13.766680    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:13.769295 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:13.769209    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 22:20:13.769750 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:13.769622    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79" podUID="9759df3a-c1a8-40e6-a087-dfd96b243c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 22:20:14.457153 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:14.457121    4889 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:20:14.457361 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:14.457176    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp" podUID="b69c09df-8ced-418f-92f4-6c9ead312931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 22:20:14.750295 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:14.750265    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 22:20:14.750463 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:14.750324    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79" podUID="9759df3a-c1a8-40e6-a087-dfd96b243c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 22:20:15.457646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:15.457611    4889 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:20:15.458184 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:15.457658    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp" podUID="b69c09df-8ced-418f-92f4-6c9ead312931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 22:20:15.543602 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:15.538688    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:15.753506 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:15.753481    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:15.754005 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:15.753994    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:15.754720 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:15.754696    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:20:16.461768 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:16.461733    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:16.462212 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:16.462155    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:16.462987 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:16.462966    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 22:20:17.403593 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:17.403260    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:19.371875 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:19.371835    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:21.453954 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:21.453917    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:22.308047 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.308015    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"]
Oct 30 22:20:22.313361 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.313343    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.340494 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.340467    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"]
Oct 30 22:20:22.411953 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.411919    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f63c047-c985-45ed-92b5-3fed61c84385-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.411953 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.411956    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.412144 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.412063    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.412180 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.412144    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.412256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.412217    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.412298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.412264    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.412298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.412282    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f63c047-c985-45ed-92b5-3fed61c84385-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.412362 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.412303    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f63c047-c985-45ed-92b5-3fed61c84385-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.412362 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.412346    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfzxr\" (UniqueName: \"kubernetes.io/projected/4f63c047-c985-45ed-92b5-3fed61c84385-kube-api-access-jfzxr\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.513087 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.513046    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.513608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.513108    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.513608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.513204    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.513608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.513250    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.513608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.513278    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f63c047-c985-45ed-92b5-3fed61c84385-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.513608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.513405    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f63c047-c985-45ed-92b5-3fed61c84385-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.513608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.513448    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfzxr\" (UniqueName: \"kubernetes.io/projected/4f63c047-c985-45ed-92b5-3fed61c84385-kube-api-access-jfzxr\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.513608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.513551    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f63c047-c985-45ed-92b5-3fed61c84385-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.513608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.513581    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.513608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.513594    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.514075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.513688    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.514075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.513922    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.514075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.514044    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.514464 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.514445    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f63c047-c985-45ed-92b5-3fed61c84385-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.515976 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.515933    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.516578 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.516548    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f63c047-c985-45ed-92b5-3fed61c84385-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.536444 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.536423    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f63c047-c985-45ed-92b5-3fed61c84385-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.538259 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.538219    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfzxr\" (UniqueName: \"kubernetes.io/projected/4f63c047-c985-45ed-92b5-3fed61c84385-kube-api-access-jfzxr\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.553407 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.553383    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"]
Oct 30 22:20:22.558523 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.558468    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:20:22.574567 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.574531    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-09d0d5\"/\"distinct-w0-09d0d5-kserve-self-signed-certs\""
Oct 30 22:20:22.578943 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.578920    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"]
Oct 30 22:20:22.614758 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.614724    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/009fde76-6593-492a-8c91-b566aa3f327a-tls-certs\") pod \"distinct-w0-09d0d5-kserve-5db7449545-qlgvc\" (UID: \"009fde76-6593-492a-8c91-b566aa3f327a\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:20:22.614937 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.614904    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-model-cache\") pod \"distinct-w0-09d0d5-kserve-5db7449545-qlgvc\" (UID: \"009fde76-6593-492a-8c91-b566aa3f327a\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:20:22.614991 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.614941    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-dshm\") pod \"distinct-w0-09d0d5-kserve-5db7449545-qlgvc\" (UID: \"009fde76-6593-492a-8c91-b566aa3f327a\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:20:22.614991 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.614965    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-tmp-dir\") pod \"distinct-w0-09d0d5-kserve-5db7449545-qlgvc\" (UID: \"009fde76-6593-492a-8c91-b566aa3f327a\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:20:22.615106 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.615088    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-home\") pod \"distinct-w0-09d0d5-kserve-5db7449545-qlgvc\" (UID: \"009fde76-6593-492a-8c91-b566aa3f327a\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:20:22.615189 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.615175    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trh79\" (UniqueName: \"kubernetes.io/projected/009fde76-6593-492a-8c91-b566aa3f327a-kube-api-access-trh79\") pod \"distinct-w0-09d0d5-kserve-5db7449545-qlgvc\" (UID: \"009fde76-6593-492a-8c91-b566aa3f327a\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:20:22.627073 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.627048    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:22.682462 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.682214    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:22.717863 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.717809    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/009fde76-6593-492a-8c91-b566aa3f327a-tls-certs\") pod \"distinct-w0-09d0d5-kserve-5db7449545-qlgvc\" (UID: \"009fde76-6593-492a-8c91-b566aa3f327a\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:20:22.718030 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.717971    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-model-cache\") pod \"distinct-w0-09d0d5-kserve-5db7449545-qlgvc\" (UID: \"009fde76-6593-492a-8c91-b566aa3f327a\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:20:22.718030 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.718007    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-dshm\") pod \"distinct-w0-09d0d5-kserve-5db7449545-qlgvc\" (UID: \"009fde76-6593-492a-8c91-b566aa3f327a\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:20:22.718157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.718033    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-tmp-dir\") pod \"distinct-w0-09d0d5-kserve-5db7449545-qlgvc\" (UID: \"009fde76-6593-492a-8c91-b566aa3f327a\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:20:22.718157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.718098    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-home\") pod \"distinct-w0-09d0d5-kserve-5db7449545-qlgvc\" (UID: \"009fde76-6593-492a-8c91-b566aa3f327a\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:20:22.718267 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.718160    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trh79\" (UniqueName: \"kubernetes.io/projected/009fde76-6593-492a-8c91-b566aa3f327a-kube-api-access-trh79\") pod \"distinct-w0-09d0d5-kserve-5db7449545-qlgvc\" (UID: \"009fde76-6593-492a-8c91-b566aa3f327a\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:20:22.720038 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.719665    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-home\") pod \"distinct-w0-09d0d5-kserve-5db7449545-qlgvc\" (UID: \"009fde76-6593-492a-8c91-b566aa3f327a\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:20:22.720038 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.719697    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-tmp-dir\") pod \"distinct-w0-09d0d5-kserve-5db7449545-qlgvc\" (UID: \"009fde76-6593-492a-8c91-b566aa3f327a\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:20:22.720345 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.720303    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-model-cache\") pod \"distinct-w0-09d0d5-kserve-5db7449545-qlgvc\" (UID: \"009fde76-6593-492a-8c91-b566aa3f327a\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:20:22.724996 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.724974    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-dshm\") pod \"distinct-w0-09d0d5-kserve-5db7449545-qlgvc\" (UID: \"009fde76-6593-492a-8c91-b566aa3f327a\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:20:22.732351 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.732313    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/009fde76-6593-492a-8c91-b566aa3f327a-tls-certs\") pod \"distinct-w0-09d0d5-kserve-5db7449545-qlgvc\" (UID: \"009fde76-6593-492a-8c91-b566aa3f327a\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:20:22.759892 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.749497    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trh79\" (UniqueName: \"kubernetes.io/projected/009fde76-6593-492a-8c91-b566aa3f327a-kube-api-access-trh79\") pod \"distinct-w0-09d0d5-kserve-5db7449545-qlgvc\" (UID: \"009fde76-6593-492a-8c91-b566aa3f327a\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:20:22.877506 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.877415    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:20:22.961131 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:20:22.961099    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f63c047_c985_45ed_92b5_3fed61c84385.slice/crio-ee5321f714e07cd83e8a6a38c4613efa3fd6476021925986a8a68ec911352e5a WatchSource:0}: Error finding container ee5321f714e07cd83e8a6a38c4613efa3fd6476021925986a8a68ec911352e5a: Status 404 returned error can't find the container with id ee5321f714e07cd83e8a6a38c4613efa3fd6476021925986a8a68ec911352e5a
Oct 30 22:20:22.962986 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.962962    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"]
Oct 30 22:20:22.964984 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.964928    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:20:22.965113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.965004    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:20:22.965113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:22.965044    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:20:23.191319 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:20:23.191288    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod009fde76_6593_492a_8c91_b566aa3f327a.slice/crio-e544b1da1ff67e90e3efdbd91b2e4818704690c6a7f731c70f0321eab63176ca WatchSource:0}: Error finding container e544b1da1ff67e90e3efdbd91b2e4818704690c6a7f731c70f0321eab63176ca: Status 404 returned error can't find the container with id e544b1da1ff67e90e3efdbd91b2e4818704690c6a7f731c70f0321eab63176ca
Oct 30 22:20:23.191743 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:23.191722    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"]
Oct 30 22:20:23.221054 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:23.221023    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" event={"ID":"009fde76-6593-492a-8c91-b566aa3f327a","Type":"ContainerStarted","Data":"e544b1da1ff67e90e3efdbd91b2e4818704690c6a7f731c70f0321eab63176ca"}
Oct 30 22:20:23.223675 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:23.223647    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k" event={"ID":"4f63c047-c985-45ed-92b5-3fed61c84385","Type":"ContainerStarted","Data":"967d05a6cea8cfdee6de2d406efa400eae39299234f916036c8e6f633b13d631"}
Oct 30 22:20:23.223785 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:23.223710    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k" event={"ID":"4f63c047-c985-45ed-92b5-3fed61c84385","Type":"ContainerStarted","Data":"ee5321f714e07cd83e8a6a38c4613efa3fd6476021925986a8a68ec911352e5a"}
Oct 30 22:20:23.270354 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:23.269421    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k" podStartSLOduration=1.269404682 podStartE2EDuration="1.269404682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:23.266820746 +0000 UTC m=+808.623187060" watchObservedRunningTime="2026-10-30 22:20:23.269404682 +0000 UTC m=+808.625770998"
Oct 30 22:20:23.379346 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:23.375114    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:23.632490 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:23.632385    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:24.238928 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:24.238890    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" event={"ID":"009fde76-6593-492a-8c91-b566aa3f327a","Type":"ContainerStarted","Data":"ed7044681b37bb733a06c65453d32e863d61664739335c4238e2da725df14571"}
Oct 30 22:20:24.279367 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:24.279314    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podStartSLOduration=2.279298973 podStartE2EDuration="2.279298973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:24.277982181 +0000 UTC m=+809.634348495" watchObservedRunningTime="2026-10-30 22:20:24.279298973 +0000 UTC m=+809.635665286"
Oct 30 22:20:24.635572 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:24.635532    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 30 22:20:24.636107 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:24.635602    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k" podUID="4f63c047-c985-45ed-92b5-3fed61c84385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 30 22:20:24.877846 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:24.877808    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:20:25.396896 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:25.391467    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:25.634650 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:25.634606    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 30 22:20:25.634820 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:25.634670    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k" podUID="4f63c047-c985-45ed-92b5-3fed61c84385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 30 22:20:26.631249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:26.630614    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 30 22:20:26.631249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:26.630676    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k" podUID="4f63c047-c985-45ed-92b5-3fed61c84385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 30 22:20:27.270253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:27.267879    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:27.632337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:27.630661    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 30 22:20:27.632337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:27.630736    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k" podUID="4f63c047-c985-45ed-92b5-3fed61c84385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 30 22:20:28.627635 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:28.627597    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 30 22:20:28.627945 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:28.627917    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k" podUID="4f63c047-c985-45ed-92b5-3fed61c84385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 30 22:20:29.051200 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:29.051168    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:29.702433 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:29.702400    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:29.703360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:29.703335    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:29.704436 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:29.704410    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:20:30.584345 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:30.584314    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:30.863217 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:30.863129    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:32.411990 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.408393    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"]
Oct 30 22:20:32.438476 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.438439    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"]
Oct 30 22:20:32.438713 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.438577    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:20:32.454666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.454265    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-54114e\"/\"openshift-service-ca.crt\""
Oct 30 22:20:32.469203 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.469176    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-54114e\"/\"kube-root-ca.crt\""
Oct 30 22:20:32.469533 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.469516    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-54114e\"/\"default-dockercfg-nnn5d\""
Oct 30 22:20:32.469840 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.469811    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-54114e\"/\"distinct-w5-54114e-kserve-self-signed-certs\""
Oct 30 22:20:32.570314 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.570072    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x6km\" (UniqueName: \"kubernetes.io/projected/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-kube-api-access-8x6km\") pod \"distinct-w5-54114e-kserve-6995667bf6-hjvzd\" (UID: \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:20:32.570700 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.570674    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-home\") pod \"distinct-w5-54114e-kserve-6995667bf6-hjvzd\" (UID: \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:20:32.570918 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.570897    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-dshm\") pod \"distinct-w5-54114e-kserve-6995667bf6-hjvzd\" (UID: \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:20:32.571806 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.571277    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:32.571981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.571961    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-model-cache\") pod \"distinct-w5-54114e-kserve-6995667bf6-hjvzd\" (UID: \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:20:32.572142 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.572118    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-tls-certs\") pod \"distinct-w5-54114e-kserve-6995667bf6-hjvzd\" (UID: \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:20:32.572254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.572159    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-tmp-dir\") pod \"distinct-w5-54114e-kserve-6995667bf6-hjvzd\" (UID: \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:20:32.677374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.677245    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-home\") pod \"distinct-w5-54114e-kserve-6995667bf6-hjvzd\" (UID: \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:20:32.678179 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.678154    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-dshm\") pod \"distinct-w5-54114e-kserve-6995667bf6-hjvzd\" (UID: \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:20:32.678498 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.678478    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-model-cache\") pod \"distinct-w5-54114e-kserve-6995667bf6-hjvzd\" (UID: \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:20:32.678636 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.678621    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-tls-certs\") pod \"distinct-w5-54114e-kserve-6995667bf6-hjvzd\" (UID: \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:20:32.678755 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.678740    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-tmp-dir\") pod \"distinct-w5-54114e-kserve-6995667bf6-hjvzd\" (UID: \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:20:32.678850 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.678835    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-model-cache\") pod \"distinct-w5-54114e-kserve-6995667bf6-hjvzd\" (UID: \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:20:32.678951 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.678914    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x6km\" (UniqueName: \"kubernetes.io/projected/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-kube-api-access-8x6km\") pod \"distinct-w5-54114e-kserve-6995667bf6-hjvzd\" (UID: \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:20:32.679471 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.679446    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-tmp-dir\") pod \"distinct-w5-54114e-kserve-6995667bf6-hjvzd\" (UID: \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:20:32.679553 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.678024    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-home\") pod \"distinct-w5-54114e-kserve-6995667bf6-hjvzd\" (UID: \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:20:32.695936 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.695905    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-dshm\") pod \"distinct-w5-54114e-kserve-6995667bf6-hjvzd\" (UID: \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:20:32.711855 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.710849    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x6km\" (UniqueName: \"kubernetes.io/projected/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-kube-api-access-8x6km\") pod \"distinct-w5-54114e-kserve-6995667bf6-hjvzd\" (UID: \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:20:32.736936 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.736897    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-tls-certs\") pod \"distinct-w5-54114e-kserve-6995667bf6-hjvzd\" (UID: \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:20:32.759694 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.759661    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:32.792555 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.792530    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:20:32.880602 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.879657    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:20:32.917672 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:32.917277    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:20:33.206121 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:20:33.206094    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd0ddc98_49ec_4ca9_ba5d_1db4b785b186.slice/crio-eb152f66dfb86ecc9845d82d560b3c2b1544ce9ccc95a3a268713b2df5f008a1 WatchSource:0}: Error finding container eb152f66dfb86ecc9845d82d560b3c2b1544ce9ccc95a3a268713b2df5f008a1: Status 404 returned error can't find the container with id eb152f66dfb86ecc9845d82d560b3c2b1544ce9ccc95a3a268713b2df5f008a1
Oct 30 22:20:33.206249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:33.206124    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"]
Oct 30 22:20:33.317266 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:33.317211    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" event={"ID":"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186","Type":"ContainerStarted","Data":"eccf8de25ed87a3c14921ff096bd70fd5580728bb2ee9c3e82014412963c69b1"}
Oct 30 22:20:33.317436 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:33.317279    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" event={"ID":"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186","Type":"ContainerStarted","Data":"eb152f66dfb86ecc9845d82d560b3c2b1544ce9ccc95a3a268713b2df5f008a1"}
Oct 30 22:20:33.329323 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:33.329299    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:20:33.358008 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:33.357959    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podStartSLOduration=1.357946818 podStartE2EDuration="1.357946818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:33.355684112 +0000 UTC m=+818.712050426" watchObservedRunningTime="2026-10-30 22:20:33.357946818 +0000 UTC m=+818.714313131"
Oct 30 22:20:33.794496 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:33.794471    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:20:34.606137 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:34.606098    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:36.344248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:36.339456    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:38.096907 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:38.096860    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:39.641938 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:39.641898    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:40.014247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:40.004639    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:40.301010 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:20:40.300929    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:20:41.740654 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:41.740613    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:42.794614 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:42.794585    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:20:42.806604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:42.806583    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:20:43.419912 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:43.419887    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:20:43.537249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:43.502301    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:45.148171 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:45.126411    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:45.873105 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:45.873070    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:46.828137 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:46.828052    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:48.536328 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:48.536288    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:49.070787 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.070749    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"]
Oct 30 22:20:49.076027 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.076003    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:20:49.090828 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.090800    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"]
Oct 30 22:20:49.099592 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.099567    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-421583\"/\"distinct-2-w2-421583-kserve-self-signed-certs\""
Oct 30 22:20:49.099740 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.099655    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-421583\"/\"default-dockercfg-vhmms\""
Oct 30 22:20:49.102951 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.102932    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-421583\"/\"openshift-service-ca.crt\""
Oct 30 22:20:49.103093 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.102975    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-421583\"/\"kube-root-ca.crt\""
Oct 30 22:20:49.128155 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.128134    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-home\") pod \"distinct-2-w2-421583-kserve-565d4d7cc-vlllf\" (UID: \"da44d50b-144e-4f4a-bd5b-195b78448b0f\") " pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:20:49.128289 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.128271    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swwfs\" (UniqueName: \"kubernetes.io/projected/da44d50b-144e-4f4a-bd5b-195b78448b0f-kube-api-access-swwfs\") pod \"distinct-2-w2-421583-kserve-565d4d7cc-vlllf\" (UID: \"da44d50b-144e-4f4a-bd5b-195b78448b0f\") " pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:20:49.128334 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.128308    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-dshm\") pod \"distinct-2-w2-421583-kserve-565d4d7cc-vlllf\" (UID: \"da44d50b-144e-4f4a-bd5b-195b78448b0f\") " pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:20:49.128371 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.128346    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-model-cache\") pod \"distinct-2-w2-421583-kserve-565d4d7cc-vlllf\" (UID: \"da44d50b-144e-4f4a-bd5b-195b78448b0f\") " pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:20:49.128513 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.128497    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-tmp-dir\") pod \"distinct-2-w2-421583-kserve-565d4d7cc-vlllf\" (UID: \"da44d50b-144e-4f4a-bd5b-195b78448b0f\") " pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:20:49.128547 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.128522    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da44d50b-144e-4f4a-bd5b-195b78448b0f-tls-certs\") pod \"distinct-2-w2-421583-kserve-565d4d7cc-vlllf\" (UID: \"da44d50b-144e-4f4a-bd5b-195b78448b0f\") " pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:20:49.190051 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.190022    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:49.229685 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.229661    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-tmp-dir\") pod \"distinct-2-w2-421583-kserve-565d4d7cc-vlllf\" (UID: \"da44d50b-144e-4f4a-bd5b-195b78448b0f\") " pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:20:49.229875 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.229693    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da44d50b-144e-4f4a-bd5b-195b78448b0f-tls-certs\") pod \"distinct-2-w2-421583-kserve-565d4d7cc-vlllf\" (UID: \"da44d50b-144e-4f4a-bd5b-195b78448b0f\") " pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:20:49.229875 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.229734    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-home\") pod \"distinct-2-w2-421583-kserve-565d4d7cc-vlllf\" (UID: \"da44d50b-144e-4f4a-bd5b-195b78448b0f\") " pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:20:49.229875 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.229860    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swwfs\" (UniqueName: \"kubernetes.io/projected/da44d50b-144e-4f4a-bd5b-195b78448b0f-kube-api-access-swwfs\") pod \"distinct-2-w2-421583-kserve-565d4d7cc-vlllf\" (UID: \"da44d50b-144e-4f4a-bd5b-195b78448b0f\") " pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:20:49.230005 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.229895    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-dshm\") pod \"distinct-2-w2-421583-kserve-565d4d7cc-vlllf\" (UID: \"da44d50b-144e-4f4a-bd5b-195b78448b0f\") " pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:20:49.230005 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.229947    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-model-cache\") pod \"distinct-2-w2-421583-kserve-565d4d7cc-vlllf\" (UID: \"da44d50b-144e-4f4a-bd5b-195b78448b0f\") " pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:20:49.230122 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.230102    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-tmp-dir\") pod \"distinct-2-w2-421583-kserve-565d4d7cc-vlllf\" (UID: \"da44d50b-144e-4f4a-bd5b-195b78448b0f\") " pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:20:49.230176 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.230163    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-home\") pod \"distinct-2-w2-421583-kserve-565d4d7cc-vlllf\" (UID: \"da44d50b-144e-4f4a-bd5b-195b78448b0f\") " pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:20:49.230285 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.230268    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-model-cache\") pod \"distinct-2-w2-421583-kserve-565d4d7cc-vlllf\" (UID: \"da44d50b-144e-4f4a-bd5b-195b78448b0f\") " pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:20:49.232164 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.232143    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-dshm\") pod \"distinct-2-w2-421583-kserve-565d4d7cc-vlllf\" (UID: \"da44d50b-144e-4f4a-bd5b-195b78448b0f\") " pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:20:49.232543 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.232528    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da44d50b-144e-4f4a-bd5b-195b78448b0f-tls-certs\") pod \"distinct-2-w2-421583-kserve-565d4d7cc-vlllf\" (UID: \"da44d50b-144e-4f4a-bd5b-195b78448b0f\") " pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:20:49.250160 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.250137    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swwfs\" (UniqueName: \"kubernetes.io/projected/da44d50b-144e-4f4a-bd5b-195b78448b0f-kube-api-access-swwfs\") pod \"distinct-2-w2-421583-kserve-565d4d7cc-vlllf\" (UID: \"da44d50b-144e-4f4a-bd5b-195b78448b0f\") " pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:20:49.394133 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.394052    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:20:49.701626 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:20:49.701596    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda44d50b_144e_4f4a_bd5b_195b78448b0f.slice/crio-9bdd22b29aa43e2eb86eb55a6bd1f6e2000bfbece2e7c6eeeefd66d9bee26527 WatchSource:0}: Error finding container 9bdd22b29aa43e2eb86eb55a6bd1f6e2000bfbece2e7c6eeeefd66d9bee26527: Status 404 returned error can't find the container with id 9bdd22b29aa43e2eb86eb55a6bd1f6e2000bfbece2e7c6eeeefd66d9bee26527
Oct 30 22:20:49.702303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:49.702280    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"]
Oct 30 22:20:50.261841 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:50.261791    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:50.492273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:50.485341    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" event={"ID":"da44d50b-144e-4f4a-bd5b-195b78448b0f","Type":"ContainerStarted","Data":"1e92b74aeb385c44c95f1d7b3ff4f3cf0ffce40a856436a3bf727e73d4b7050e"}
Oct 30 22:20:50.492273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:50.485398    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" event={"ID":"da44d50b-144e-4f4a-bd5b-195b78448b0f","Type":"ContainerStarted","Data":"9bdd22b29aa43e2eb86eb55a6bd1f6e2000bfbece2e7c6eeeefd66d9bee26527"}
Oct 30 22:20:50.544821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:50.544702    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podStartSLOduration=1.54468425 podStartE2EDuration="1.54468425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:50.541848391 +0000 UTC m=+835.898214706" watchObservedRunningTime="2026-10-30 22:20:50.54468425 +0000 UTC m=+835.901050556"
Oct 30 22:20:51.394322 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:51.394291    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:20:52.335245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:52.335024    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:52.936847 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:52.936813    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"]
Oct 30 22:20:52.941032 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:52.941011    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:52.983945 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:52.983911    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"]
Oct 30 22:20:53.009262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.009196    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.009489 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.009311    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89d5a590-0ac8-4be8-a791-942b739fc869-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.009489 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.009424    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d5a590-0ac8-4be8-a791-942b739fc869-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.009489 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.009456    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p48px\" (UniqueName: \"kubernetes.io/projected/89d5a590-0ac8-4be8-a791-942b739fc869-kube-api-access-p48px\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.009675 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.009570    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.009675 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.009604    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.009791 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.009744    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d5a590-0ac8-4be8-a791-942b739fc869-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.009861 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.009827    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.009918 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.009870    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.110725 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.110693    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.110725 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.110731    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.110947 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.110772    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d5a590-0ac8-4be8-a791-942b739fc869-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.110947 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.110803    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.110947 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.110851    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.110947 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.110918    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.111166 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.110959    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89d5a590-0ac8-4be8-a791-942b739fc869-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.111166 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.111063    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d5a590-0ac8-4be8-a791-942b739fc869-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.111166 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.111100    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p48px\" (UniqueName: \"kubernetes.io/projected/89d5a590-0ac8-4be8-a791-942b739fc869-kube-api-access-p48px\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.111166 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.111133    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.111469 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.111364    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.111541 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.111524    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.111601 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.111564    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.111657 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.111607    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d5a590-0ac8-4be8-a791-942b739fc869-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.113344 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.113323    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.113929 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.113908    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d5a590-0ac8-4be8-a791-942b739fc869-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.130804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.130778    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89d5a590-0ac8-4be8-a791-942b739fc869-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.131384 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.131363    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p48px\" (UniqueName: \"kubernetes.io/projected/89d5a590-0ac8-4be8-a791-942b739fc869-kube-api-access-p48px\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.257641 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.257616    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:53.579035 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:20:53.579005    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d5a590_0ac8_4be8_a791_942b739fc869.slice/crio-7ca8f4fc49635152a0a59d652fc7bc4d0830c528d01da53705285e8eea82cb09 WatchSource:0}: Error finding container 7ca8f4fc49635152a0a59d652fc7bc4d0830c528d01da53705285e8eea82cb09: Status 404 returned error can't find the container with id 7ca8f4fc49635152a0a59d652fc7bc4d0830c528d01da53705285e8eea82cb09
Oct 30 22:20:53.579491 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.579470    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"]
Oct 30 22:20:53.581559 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.581531    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:20:53.581614 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.581595    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:20:53.581653 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:53.581621    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:20:54.144082 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:54.144049    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:54.522560 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:54.517197    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh" event={"ID":"89d5a590-0ac8-4be8-a791-942b739fc869","Type":"ContainerStarted","Data":"4a986469abd761f96f6a05beb519ef3ece3e499962f1db141cc3ac17f0a8b90c"}
Oct 30 22:20:54.522560 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:54.517292    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh" event={"ID":"89d5a590-0ac8-4be8-a791-942b739fc869","Type":"ContainerStarted","Data":"7ca8f4fc49635152a0a59d652fc7bc4d0830c528d01da53705285e8eea82cb09"}
Oct 30 22:20:54.589408 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:54.589359    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh" podStartSLOduration=2.589344638 podStartE2EDuration="2.589344638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:54.575724782 +0000 UTC m=+839.932091095" watchObservedRunningTime="2026-10-30 22:20:54.589344638 +0000 UTC m=+839.945710952"
Oct 30 22:20:55.258623 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:55.258584    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:20:55.260452 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:55.260428    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:20:55.260555 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:55.260478    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh" podUID="89d5a590-0ac8-4be8-a791-942b739fc869" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 22:20:55.584197 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:55.584101    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:56.049258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:56.048817    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:56.258195 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:56.258160    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:20:56.258402 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:56.258212    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh" podUID="89d5a590-0ac8-4be8-a791-942b739fc869" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 22:20:57.258884 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:57.258852    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:20:57.259325 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:57.258904    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh" podUID="89d5a590-0ac8-4be8-a791-942b739fc869" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 22:20:57.869288 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:57.866527    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:58.258899 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:58.258861    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:20:58.259352 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:58.258914    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh" podUID="89d5a590-0ac8-4be8-a791-942b739fc869" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 22:20:58.736929 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:58.736843    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:59.257957 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:59.257923    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:20:59.258142 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:59.257980    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh" podUID="89d5a590-0ac8-4be8-a791-942b739fc869" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 22:20:59.394586 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:59.394558    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:20:59.407176 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:59.407147    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:20:59.583841 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:59.583760    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:20:59.865552 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:20:59.865474    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:00.258393 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:00.258365    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:21:00.258556 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:00.258413    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh" podUID="89d5a590-0ac8-4be8-a791-942b739fc869" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 22:21:01.262786 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:01.262752    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:21:01.263195 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:01.263166    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:21:01.263856 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:01.263833    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:21:01.658170 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:01.658077    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:03.448061 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:03.448025    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:05.230947 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:05.227329    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:07.032377 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.032343    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:07.671926 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.671894    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"]
Oct 30 22:21:07.677425 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.677406    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.699093 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.699069    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"]
Oct 30 22:21:07.769674 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.769643    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.769846 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.769684    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.769846 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.769728    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.769921 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.769863    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.769921 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.769903    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.769989 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.769934    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.769989 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.769969    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.770049 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.769995    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbtll\" (UniqueName: \"kubernetes.io/projected/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-kube-api-access-sbtll\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.770049 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.770035    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.871312 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.871280    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.871493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.871321    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.871493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.871340    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.871493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.871355    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbtll\" (UniqueName: \"kubernetes.io/projected/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-kube-api-access-sbtll\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.871493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.871393    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.871493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.871472    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.871770 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.871512    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.871770 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.871554    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.871770 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.871746    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.871941 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.871827    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.871941 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.871896    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.871941 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.871930    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.872183 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.872142    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.872335 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.872288    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.874175 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.874154    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.874640 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.874624    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.891062 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.891037    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbtll\" (UniqueName: \"kubernetes.io/projected/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-kube-api-access-sbtll\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.891211 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.891193    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:07.991604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:07.991584    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:08.310686 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:21:08.310653    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6c9299_a77c_42a8_9b81_67f84aa9f22d.slice/crio-72f9f01daf954a2c22e650426f108a448111de4c08c9fe59d0fee4832dbc7322 WatchSource:0}: Error finding container 72f9f01daf954a2c22e650426f108a448111de4c08c9fe59d0fee4832dbc7322: Status 404 returned error can't find the container with id 72f9f01daf954a2c22e650426f108a448111de4c08c9fe59d0fee4832dbc7322
Oct 30 22:21:08.311333 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:08.311313    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"]
Oct 30 22:21:08.313198 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:08.313170    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:21:08.313275 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:08.313249    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:21:08.313313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:08.313281    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:21:08.624698 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:08.624597    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9" event={"ID":"ac6c9299-a77c-42a8-9b81-67f84aa9f22d","Type":"ContainerStarted","Data":"2f9ba3d90be6b7aac2a49d41918ab45c2b0a4e6158c17461b425d1218f5901f1"}
Oct 30 22:21:08.624698 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:08.624660    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9" event={"ID":"ac6c9299-a77c-42a8-9b81-67f84aa9f22d","Type":"ContainerStarted","Data":"72f9f01daf954a2c22e650426f108a448111de4c08c9fe59d0fee4832dbc7322"}
Oct 30 22:21:08.693976 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:08.693925    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9" podStartSLOduration=1.693910161 podStartE2EDuration="1.693910161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:08.680465353 +0000 UTC m=+854.036831666" watchObservedRunningTime="2026-10-30 22:21:08.693910161 +0000 UTC m=+854.050276475"
Oct 30 22:21:08.861264 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:08.860138    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:08.992207 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:08.992182    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:09.993388 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:09.993359    4889 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:21:09.993887 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:09.993408    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9" podUID="ac6c9299-a77c-42a8-9b81-67f84aa9f22d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 22:21:10.301446 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:21:10.301354    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:21:10.629495 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:10.618798    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:10.992857 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:10.992831    4889 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:21:10.993018 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:10.992879    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9" podUID="ac6c9299-a77c-42a8-9b81-67f84aa9f22d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 22:21:11.992504 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:11.992469    4889 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:21:11.992924 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:11.992533    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9" podUID="ac6c9299-a77c-42a8-9b81-67f84aa9f22d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 22:21:12.435783 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:12.435675    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:12.992396 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:12.992361    4889 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:21:12.993065 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:12.992653    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9" podUID="ac6c9299-a77c-42a8-9b81-67f84aa9f22d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 22:21:14.014896 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:14.014857    4889 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:21:14.029530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:14.029491    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9" podUID="ac6c9299-a77c-42a8-9b81-67f84aa9f22d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 22:21:14.309308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:14.299092    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:14.434259 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:14.430316    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:14.569872 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:14.569658    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:14.645365 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:14.645314    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:14.992597 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:14.992576    4889 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:21:14.992781 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:14.992625    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9" podUID="ac6c9299-a77c-42a8-9b81-67f84aa9f22d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 22:21:15.370109 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:15.370020    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:16.330500 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:16.330470    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:16.330821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:16.330796    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:16.331715 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:16.331694    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:21:17.145386 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:17.145355    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:18.941251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:18.932810    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:20.629248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:20.629076    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:22.350051 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.350010    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:22.617910 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.617815    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"]
Oct 30 22:21:22.622453 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.622431    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.640553 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.640522    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"]
Oct 30 22:21:22.645465 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.645442    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.654001 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.653978    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"]
Oct 30 22:21:22.672390 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.672364    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"]
Oct 30 22:21:22.770344 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.770309    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.770344 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.770348    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.770571 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.770370    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.770571 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.770544    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.770657 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.770577    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.770704 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.770690    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.770748 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.770711    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.770785 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.770746    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.770845 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.770829    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.770880 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.770870    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mfmw\" (UniqueName: \"kubernetes.io/projected/69ae6469-721f-4879-bd5f-132fbc422276-kube-api-access-8mfmw\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.770923 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.770914    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.770959 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.770931    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ae6469-721f-4879-bd5f-132fbc422276-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.770959 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.770952    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.771040 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.770973    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69ae6469-721f-4879-bd5f-132fbc422276-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.771084 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.771056    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.771084 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.771077    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdbdc\" (UniqueName: \"kubernetes.io/projected/6ad3d3d8-c760-4b47-925b-aa2c80d43883-kube-api-access-jdbdc\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.771170 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.771120    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ae6469-721f-4879-bd5f-132fbc422276-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.771218 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.771177    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.872493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.872408    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ae6469-721f-4879-bd5f-132fbc422276-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.872493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.872454    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.872493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.872488    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.872797 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.872515    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.872797 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.872544    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.872797 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.872615    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.872797 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.872640    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.872797 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.872749    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.872797 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.872775    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.873099 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.872835    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.873099 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.872891    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.873099 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.872915    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mfmw\" (UniqueName: \"kubernetes.io/projected/69ae6469-721f-4879-bd5f-132fbc422276-kube-api-access-8mfmw\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.873099 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.872930    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.873099 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.872957    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.873099 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.872972    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.873099 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.872984    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ae6469-721f-4879-bd5f-132fbc422276-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.873099 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.873042    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.873099 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.873086    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69ae6469-721f-4879-bd5f-132fbc422276-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.873599 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.873137    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.873599 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.873169    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdbdc\" (UniqueName: \"kubernetes.io/projected/6ad3d3d8-c760-4b47-925b-aa2c80d43883-kube-api-access-jdbdc\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.873599 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.873427    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.873599 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.873447    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ae6469-721f-4879-bd5f-132fbc422276-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.873599 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.873507    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.874068 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.873962    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.874180 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.874157    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.874397 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.874375    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.874504 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.874404    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.874604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.874583    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.875573 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.875546    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.875731 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.875708    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.875864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.875844    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ae6469-721f-4879-bd5f-132fbc422276-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.875904 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.875865    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.893249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.893203    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.893479 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.893457    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mfmw\" (UniqueName: \"kubernetes.io/projected/69ae6469-721f-4879-bd5f-132fbc422276-kube-api-access-8mfmw\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.893947 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.893924    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdbdc\" (UniqueName: \"kubernetes.io/projected/6ad3d3d8-c760-4b47-925b-aa2c80d43883-kube-api-access-jdbdc\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.894349 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.894328    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69ae6469-721f-4879-bd5f-132fbc422276-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:22.941704 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.941673    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:22.964184 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:22.964158    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:23.269033 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:21:23.269002    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad3d3d8_c760_4b47_925b_aa2c80d43883.slice/crio-cd1a45e2d9c411f98469c0d67b1bd23bc6a8e68891e9f802fa3cfcd06ae90022 WatchSource:0}: Error finding container cd1a45e2d9c411f98469c0d67b1bd23bc6a8e68891e9f802fa3cfcd06ae90022: Status 404 returned error can't find the container with id cd1a45e2d9c411f98469c0d67b1bd23bc6a8e68891e9f802fa3cfcd06ae90022
Oct 30 22:21:23.270085 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:23.270055    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"]
Oct 30 22:21:23.271963 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:23.271925    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:21:23.272033 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:23.272008    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:21:23.272067 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:23.272051    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:21:23.309186 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:23.309159    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"]
Oct 30 22:21:23.311307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:23.311275    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:21:23.311425 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:23.311358    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:21:23.311425 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:23.311404    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:21:23.835385 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:23.835348    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh" event={"ID":"69ae6469-721f-4879-bd5f-132fbc422276","Type":"ContainerStarted","Data":"b3ebf97f820d7239599f3a535efaddba67c88d9055561d5a53909d00eeee2aae"}
Oct 30 22:21:23.835864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:23.835421    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh" event={"ID":"69ae6469-721f-4879-bd5f-132fbc422276","Type":"ContainerStarted","Data":"f865c9810e467ed07a1246cd277d1c0fd1955012136d0bfc405b59b50ce48dd0"}
Oct 30 22:21:23.837119 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:23.837092    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr" event={"ID":"6ad3d3d8-c760-4b47-925b-aa2c80d43883","Type":"ContainerStarted","Data":"ffae4b7cddb321b84e337d0db558a30d8b26430db974accef70a3c4ce893d01c"}
Oct 30 22:21:23.837295 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:23.837137    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr" event={"ID":"6ad3d3d8-c760-4b47-925b-aa2c80d43883","Type":"ContainerStarted","Data":"cd1a45e2d9c411f98469c0d67b1bd23bc6a8e68891e9f802fa3cfcd06ae90022"}
Oct 30 22:21:23.893953 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:23.893904    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh" podStartSLOduration=1.893887539 podStartE2EDuration="1.893887539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:23.893617923 +0000 UTC m=+869.249984238" watchObservedRunningTime="2026-10-30 22:21:23.893887539 +0000 UTC m=+869.250253856"
Oct 30 22:21:23.937670 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:23.937623    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr" podStartSLOduration=1.937609683 podStartE2EDuration="1.937609683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:23.935659122 +0000 UTC m=+869.292025446" watchObservedRunningTime="2026-10-30 22:21:23.937609683 +0000 UTC m=+869.293975998"
Oct 30 22:21:23.942150 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:23.942127    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:23.964333 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:23.964311    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:24.272249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:24.265180    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:24.948252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:24.945700    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:21:24.948252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:24.945785    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr" podUID="6ad3d3d8-c760-4b47-925b-aa2c80d43883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 22:21:24.971406 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:24.971372    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:21:24.971628 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:24.971429    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh" podUID="69ae6469-721f-4879-bd5f-132fbc422276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 22:21:25.950245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:25.949566    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:21:25.950245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:25.949628    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr" podUID="6ad3d3d8-c760-4b47-925b-aa2c80d43883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 22:21:25.964839 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:25.964702    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:21:25.964839 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:25.964779    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh" podUID="69ae6469-721f-4879-bd5f-132fbc422276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 22:21:26.339074 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:26.339035    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:26.933908 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:26.933871    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:26.950099 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:26.949643    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:21:26.950099 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:26.949703    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr" podUID="6ad3d3d8-c760-4b47-925b-aa2c80d43883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 22:21:26.969568 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:26.968836    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:21:26.969568 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:26.968906    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh" podUID="69ae6469-721f-4879-bd5f-132fbc422276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 22:21:27.960934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:27.960894    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:21:27.961610 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:27.961560    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr" podUID="6ad3d3d8-c760-4b47-925b-aa2c80d43883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 22:21:27.965398 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:27.965369    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:21:27.965617 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:27.965590    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh" podUID="69ae6469-721f-4879-bd5f-132fbc422276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 22:21:28.326826 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:28.326795    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:28.942685 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:28.942649    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:21:28.942994 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:28.942967    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr" podUID="6ad3d3d8-c760-4b47-925b-aa2c80d43883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 22:21:28.966200 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:28.966167    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:21:28.966616 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:28.966252    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh" podUID="69ae6469-721f-4879-bd5f-132fbc422276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 22:21:29.942995 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:29.942957    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:21:29.943175 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:29.943043    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr" podUID="6ad3d3d8-c760-4b47-925b-aa2c80d43883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 22:21:29.964831 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:29.964798    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:21:29.965069 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:29.965043    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh" podUID="69ae6469-721f-4879-bd5f-132fbc422276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 22:21:30.309293 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:30.309257    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:30.964665 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:30.964628    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:21:30.964895 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:30.964710    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh" podUID="69ae6469-721f-4879-bd5f-132fbc422276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 22:21:31.342872 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:31.342843    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:31.344737 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:31.344714    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:31.345577 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:31.345551    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:21:31.970994 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:31.970961    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:31.971329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:31.971310    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:31.972741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:31.972722    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:21:32.383186 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:32.383146    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:32.996249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:32.996028    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:34.323363 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:34.323314    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:34.478717 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:34.478664    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:34.573356 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:34.573306    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:34.641452 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:34.641344    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:34.749360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:34.749314    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:34.842348 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:34.842291    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:34.960796 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:34.960694    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:34.985136 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:34.985093    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:35.045141 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:35.043754    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:35.384106 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:35.384070    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:37.152300 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:37.152251    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:37.899962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:37.899928    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"]
Oct 30 22:21:37.905423 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:37.905397    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:37.939726 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:37.939694    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"]
Oct 30 22:21:38.018810 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.018768    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.018988 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.018857    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.018988 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.018949    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.018988 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.018975    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx7bq\" (UniqueName: \"kubernetes.io/projected/c2a808e4-e5d8-415e-960e-1b21c1368d1b-kube-api-access-cx7bq\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.019108 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.019055    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.019108 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.019095    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.019242 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.019211    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.019289 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.019272    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.019352 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.019341    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.075803 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.075765    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"]
Oct 30 22:21:38.079953 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.079929    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.106116 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.106065    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"]
Oct 30 22:21:38.121164 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.121130    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.121377 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.121178    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.121377 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.121297    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.121377 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.121333    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f08b855f-771b-4002-9555-c7d56ba510c8-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.121535 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.121381    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.121535 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.121427    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.121535 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.121475    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.121535 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.121502    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f08b855f-771b-4002-9555-c7d56ba510c8-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.121712 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.121538    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.121712 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.121567    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.121712 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.121642    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.121712 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.121676    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f08b855f-771b-4002-9555-c7d56ba510c8-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.121888 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.121718    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.121888 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.121745    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.121888 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.121778    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx7bq\" (UniqueName: \"kubernetes.io/projected/c2a808e4-e5d8-415e-960e-1b21c1368d1b-kube-api-access-cx7bq\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.121888 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.121809    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.121888 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.121839    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.121888 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.121878    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.122138 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.121894    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.122138 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.121973    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrtqd\" (UniqueName: \"kubernetes.io/projected/f08b855f-771b-4002-9555-c7d56ba510c8-kube-api-access-nrtqd\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.122138 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.122132    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.122477 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.122457    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.124154 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.124127    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.125622 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.125584    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.126589 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.126507    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.143614 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.143566    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.143728 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.143706    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx7bq\" (UniqueName: \"kubernetes.io/projected/c2a808e4-e5d8-415e-960e-1b21c1368d1b-kube-api-access-cx7bq\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.221815 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.221732    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:38.223417 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.223381    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.223558 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.223475    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrtqd\" (UniqueName: \"kubernetes.io/projected/f08b855f-771b-4002-9555-c7d56ba510c8-kube-api-access-nrtqd\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.223652 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.223638    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.223716 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.223671    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f08b855f-771b-4002-9555-c7d56ba510c8-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.223773 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.223732    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.223845 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.223777    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f08b855f-771b-4002-9555-c7d56ba510c8-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.223904 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.223860    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.223904 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.223892    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f08b855f-771b-4002-9555-c7d56ba510c8-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.224009 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.223934    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.224064 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.224040    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.224119 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.224102    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.224428 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.224404    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.224554 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.224534    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.224605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.224544    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f08b855f-771b-4002-9555-c7d56ba510c8-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.227295 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.227276    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.227387 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.227368    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f08b855f-771b-4002-9555-c7d56ba510c8-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.243932 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.243910    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f08b855f-771b-4002-9555-c7d56ba510c8-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.244014 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.244000    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrtqd\" (UniqueName: \"kubernetes.io/projected/f08b855f-771b-4002-9555-c7d56ba510c8-kube-api-access-nrtqd\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.403541 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.403511    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:38.549471 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:21:38.549436    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2a808e4_e5d8_415e_960e_1b21c1368d1b.slice/crio-83627d818a1c83f74eb28e7eaf22ceb24a6ee9de0cea1a0c56c28eea6dc08195 WatchSource:0}: Error finding container 83627d818a1c83f74eb28e7eaf22ceb24a6ee9de0cea1a0c56c28eea6dc08195: Status 404 returned error can't find the container with id 83627d818a1c83f74eb28e7eaf22ceb24a6ee9de0cea1a0c56c28eea6dc08195
Oct 30 22:21:38.550410 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.550386    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"]
Oct 30 22:21:38.552660 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.552625    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:21:38.552753 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.552700    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:21:38.552753 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.552735    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:21:38.715130 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.715082    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"]
Oct 30 22:21:38.716330 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:21:38.716303    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf08b855f_771b_4002_9555_c7d56ba510c8.slice/crio-471a4c09707b03d4dc36a78f02ba4d2659fd9885d0cc31d40e7d9b6b99649c2d WatchSource:0}: Error finding container 471a4c09707b03d4dc36a78f02ba4d2659fd9885d0cc31d40e7d9b6b99649c2d: Status 404 returned error can't find the container with id 471a4c09707b03d4dc36a78f02ba4d2659fd9885d0cc31d40e7d9b6b99649c2d
Oct 30 22:21:38.719341 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.719306    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:21:38.719427 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.719397    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:21:38.719484 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.719468    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:21:38.939997 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:38.939859    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:39.242648 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:39.242606    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6" event={"ID":"c2a808e4-e5d8-415e-960e-1b21c1368d1b","Type":"ContainerStarted","Data":"2ea8f2e33e0c9c0fd3ad0942994dbab0cdb352410231a3f000a05d7cef9d8673"}
Oct 30 22:21:39.243142 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:39.242696    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6" event={"ID":"c2a808e4-e5d8-415e-960e-1b21c1368d1b","Type":"ContainerStarted","Data":"83627d818a1c83f74eb28e7eaf22ceb24a6ee9de0cea1a0c56c28eea6dc08195"}
Oct 30 22:21:39.244384 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:39.244358    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8" event={"ID":"f08b855f-771b-4002-9555-c7d56ba510c8","Type":"ContainerStarted","Data":"6813827c1eac5f19ef56ed93458c0c58a68ccd529f1cd2af91606a58b0d53a4f"}
Oct 30 22:21:39.244511 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:39.244405    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8" event={"ID":"f08b855f-771b-4002-9555-c7d56ba510c8","Type":"ContainerStarted","Data":"471a4c09707b03d4dc36a78f02ba4d2659fd9885d0cc31d40e7d9b6b99649c2d"}
Oct 30 22:21:39.299648 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:39.299537    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6" podStartSLOduration=2.299522677 podStartE2EDuration="2.299522677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:39.298065546 +0000 UTC m=+884.654431863" watchObservedRunningTime="2026-10-30 22:21:39.299522677 +0000 UTC m=+884.655889044"
Oct 30 22:21:39.344241 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:39.344167    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8" podStartSLOduration=1.344152569 podStartE2EDuration="1.344152569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:39.340706925 +0000 UTC m=+884.697073240" watchObservedRunningTime="2026-10-30 22:21:39.344152569 +0000 UTC m=+884.700518884"
Oct 30 22:21:39.404648 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:39.404608    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:39.412246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:39.410357    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:21:39.412246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:39.410426    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8" podUID="f08b855f-771b-4002-9555-c7d56ba510c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 22:21:40.230326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:40.230289    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:40.231100 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:40.231071    4889 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 22:21:40.231333 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:40.231307    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6" podUID="c2a808e4-e5d8-415e-960e-1b21c1368d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 22:21:40.302057 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:21:40.302020    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:21:40.404682 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:40.404637    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:21:40.404879 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:40.404696    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8" podUID="f08b855f-771b-4002-9555-c7d56ba510c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 22:21:40.968477 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:40.968440    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:41.226314 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:41.222642    4889 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 22:21:41.226314 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:41.222705    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6" podUID="c2a808e4-e5d8-415e-960e-1b21c1368d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 22:21:41.404674 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:41.404646    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:21:41.405135 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:41.405110    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8" podUID="f08b855f-771b-4002-9555-c7d56ba510c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 22:21:42.230251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:42.229668    4889 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 22:21:42.230251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:42.229732    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6" podUID="c2a808e4-e5d8-415e-960e-1b21c1368d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 22:21:42.404727 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:42.404694    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:21:42.405113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:42.404746    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8" podUID="f08b855f-771b-4002-9555-c7d56ba510c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 22:21:43.005268 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.003720    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:43.144316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.144281    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"]
Oct 30 22:21:43.188826 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.187260    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"]
Oct 30 22:21:43.188826 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.187346    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:21:43.194611 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.194046    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:43.202864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.202837    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-2b1e88\"/\"distinct-2-w1-2b1e88-kserve-self-signed-certs\""
Oct 30 22:21:43.245255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.236610    4889 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 22:21:43.267607 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.267508    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6" podUID="c2a808e4-e5d8-415e-960e-1b21c1368d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 22:21:43.324250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.320134    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-model-cache\") pod \"distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4\" (UID: \"0e99cfe5-3268-401d-b985-7634e2bac80b\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:21:43.324250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.320190    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-home\") pod \"distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4\" (UID: \"0e99cfe5-3268-401d-b985-7634e2bac80b\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:21:43.332256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.329296    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e99cfe5-3268-401d-b985-7634e2bac80b-tls-certs\") pod \"distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4\" (UID: \"0e99cfe5-3268-401d-b985-7634e2bac80b\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:21:43.332256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.329402    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ftz5\" (UniqueName: \"kubernetes.io/projected/0e99cfe5-3268-401d-b985-7634e2bac80b-kube-api-access-7ftz5\") pod \"distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4\" (UID: \"0e99cfe5-3268-401d-b985-7634e2bac80b\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:21:43.332256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.329548    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-dshm\") pod \"distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4\" (UID: \"0e99cfe5-3268-401d-b985-7634e2bac80b\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:21:43.332256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.329630    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-tmp-dir\") pod \"distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4\" (UID: \"0e99cfe5-3268-401d-b985-7634e2bac80b\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:21:43.419324 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.405604    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:21:43.419324 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.405669    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8" podUID="f08b855f-771b-4002-9555-c7d56ba510c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 22:21:43.454263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.449461    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-model-cache\") pod \"distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4\" (UID: \"0e99cfe5-3268-401d-b985-7634e2bac80b\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:21:43.454263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.449516    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-home\") pod \"distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4\" (UID: \"0e99cfe5-3268-401d-b985-7634e2bac80b\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:21:43.454263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.449545    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e99cfe5-3268-401d-b985-7634e2bac80b-tls-certs\") pod \"distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4\" (UID: \"0e99cfe5-3268-401d-b985-7634e2bac80b\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:21:43.454263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.449591    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ftz5\" (UniqueName: \"kubernetes.io/projected/0e99cfe5-3268-401d-b985-7634e2bac80b-kube-api-access-7ftz5\") pod \"distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4\" (UID: \"0e99cfe5-3268-401d-b985-7634e2bac80b\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:21:43.454263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.449681    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-dshm\") pod \"distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4\" (UID: \"0e99cfe5-3268-401d-b985-7634e2bac80b\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:21:43.454263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.449734    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-tmp-dir\") pod \"distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4\" (UID: \"0e99cfe5-3268-401d-b985-7634e2bac80b\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:21:43.454263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.450265    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-tmp-dir\") pod \"distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4\" (UID: \"0e99cfe5-3268-401d-b985-7634e2bac80b\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:21:43.485276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.477190    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-model-cache\") pod \"distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4\" (UID: \"0e99cfe5-3268-401d-b985-7634e2bac80b\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:21:43.490491 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.487303    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-home\") pod \"distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4\" (UID: \"0e99cfe5-3268-401d-b985-7634e2bac80b\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:21:43.643314 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.641552    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-dshm\") pod \"distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4\" (UID: \"0e99cfe5-3268-401d-b985-7634e2bac80b\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:21:43.680339 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.674172    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e99cfe5-3268-401d-b985-7634e2bac80b-tls-certs\") pod \"distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4\" (UID: \"0e99cfe5-3268-401d-b985-7634e2bac80b\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:21:43.680339 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.674883    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ftz5\" (UniqueName: \"kubernetes.io/projected/0e99cfe5-3268-401d-b985-7634e2bac80b-kube-api-access-7ftz5\") pod \"distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4\" (UID: \"0e99cfe5-3268-401d-b985-7634e2bac80b\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:21:43.806903 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:43.806871    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:21:44.240442 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:44.240351    4889 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 22:21:44.240732 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:44.240699    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6" podUID="c2a808e4-e5d8-415e-960e-1b21c1368d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 22:21:44.253564 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:44.253520    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"]
Oct 30 22:21:44.306386 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:44.306359    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" event={"ID":"0e99cfe5-3268-401d-b985-7634e2bac80b","Type":"ContainerStarted","Data":"a5c5615a171f68e7bae36c2aef3689bbe69993dc1da669a29b22e6dcd65adb22"}
Oct 30 22:21:44.404652 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:44.404625    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:21:44.404790 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:44.404669    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8" podUID="f08b855f-771b-4002-9555-c7d56ba510c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 22:21:45.243261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:45.241616    4889 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 22:21:45.243261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:45.241677    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6" podUID="c2a808e4-e5d8-415e-960e-1b21c1368d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 22:21:45.333257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:45.331716    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:45.356251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:45.354465    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" event={"ID":"0e99cfe5-3268-401d-b985-7634e2bac80b","Type":"ContainerStarted","Data":"ea23841308d9459ab2e480beee95231491f95a06b2621d29dd2489c61ccca6c2"}
Oct 30 22:21:45.418250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:45.413580    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:21:45.418250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:45.413636    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8" podUID="f08b855f-771b-4002-9555-c7d56ba510c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 22:21:45.807126 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:45.807097    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:21:45.991334 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:45.991299    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:46.222735 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:46.222648    4889 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 22:21:46.222735 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:46.222712    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6" podUID="c2a808e4-e5d8-415e-960e-1b21c1368d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 22:21:46.889563 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:46.889522    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:46.890555 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:46.890486    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:46.891418 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:46.891392    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:21:46.961192 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:46.961139    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podStartSLOduration=3.961120631 podStartE2EDuration="3.961120631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:45.413824216 +0000 UTC m=+890.770190532" watchObservedRunningTime="2026-10-30 22:21:46.961120631 +0000 UTC m=+892.317486946"
Oct 30 22:21:47.047856 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:47.047817    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:47.256114 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:47.251706    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:47.259888 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:47.259854    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:47.266809 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:47.266769    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:21:48.827006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:48.826972    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:51.492965 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:51.492930    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:52.378140 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.378100    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:52.545987 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.545953    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"]
Oct 30 22:21:52.550849 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.550827    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.563577 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.563550    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"]
Oct 30 22:21:52.568277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.568214    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.578670 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.578645    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"]
Oct 30 22:21:52.594587 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.594561    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"]
Oct 30 22:21:52.657712 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.657636    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqbx6\" (UniqueName: \"kubernetes.io/projected/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-kube-api-access-rqbx6\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.657876 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.657744    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.657876 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.657796    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.657876 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.657833    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.658048 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.657881    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.658048 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.657911    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.658048 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.658034    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.658206 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.658076    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.658206 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.658154    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.658372 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.658253    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.658372 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.658304    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.658475 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.658438    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.658533 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.658491    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.658578 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.658548    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.658578 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.658573    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.658670 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.658604    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.658670 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.658649    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvf6p\" (UniqueName: \"kubernetes.io/projected/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-kube-api-access-mvf6p\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.658839 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.658710    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.759904 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.759873    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqbx6\" (UniqueName: \"kubernetes.io/projected/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-kube-api-access-rqbx6\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.760080 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.759934    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.760080 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.759953    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.760080 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.759981    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.760080 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.760002    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.760080 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.760035    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.760346 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.760088    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.760346 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.760116    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.760346 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.760153    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.760346 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.760212    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.760346 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.760303    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.760582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.760354    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.760582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.760405    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.760582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.760462    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.760582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.760489    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.760582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.760535    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.760582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.760534    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.760582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.760571    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvf6p\" (UniqueName: \"kubernetes.io/projected/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-kube-api-access-mvf6p\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.760582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.760577    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.760928 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.760607    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.760928 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.760759    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.760928 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.760807    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.761042 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.760973    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.761042 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.760997    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.761042 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.761008    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.761408 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.761383    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.761408 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.761398    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.761859 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.761808    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.763005 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.762984    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.763473 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.763447    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.763667 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.763647    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.763807 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.763789    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.781468 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.781439    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.781774 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.781754    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqbx6\" (UniqueName: \"kubernetes.io/projected/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-kube-api-access-rqbx6\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.781847 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.781756    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:52.784805 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.784788    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvf6p\" (UniqueName: \"kubernetes.io/projected/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-kube-api-access-mvf6p\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.866811 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.866786    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:52.898933 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:52.898905    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:53.192483 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:21:53.192452    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408d956d_ee1c_44e8_9bc5_8d3fc72ab890.slice/crio-89caee667c866bb1e6b967622514230eff9143cd61b3c41bb16d8f4d8ea878ff WatchSource:0}: Error finding container 89caee667c866bb1e6b967622514230eff9143cd61b3c41bb16d8f4d8ea878ff: Status 404 returned error can't find the container with id 89caee667c866bb1e6b967622514230eff9143cd61b3c41bb16d8f4d8ea878ff
Oct 30 22:21:53.193454 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:53.193360    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"]
Oct 30 22:21:53.195486 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:53.195455    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:21:53.195561 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:53.195519    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:21:53.195561 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:53.195549    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:21:53.246178 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:21:53.246150    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd1e5c9d_b6f4_4326_8b97_9f0087c600c7.slice/crio-092efeb1f1d53fbbaaed80f82b05242164ed59b80fa8db85d8b8a189d731a7e5 WatchSource:0}: Error finding container 092efeb1f1d53fbbaaed80f82b05242164ed59b80fa8db85d8b8a189d731a7e5: Status 404 returned error can't find the container with id 092efeb1f1d53fbbaaed80f82b05242164ed59b80fa8db85d8b8a189d731a7e5
Oct 30 22:21:53.247157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:53.247133    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"]
Oct 30 22:21:53.248842 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:53.248811    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:21:53.248920 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:53.248877    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:21:53.248920 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:53.248906    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:21:53.481784 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:53.481686    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv" event={"ID":"408d956d-ee1c-44e8-9bc5-8d3fc72ab890","Type":"ContainerStarted","Data":"d42888dc1dab5f808a283a03d24fc1c473662a6fcecf7d7ccf3d2628808135a0"}
Oct 30 22:21:53.482076 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:53.482052    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv" event={"ID":"408d956d-ee1c-44e8-9bc5-8d3fc72ab890","Type":"ContainerStarted","Data":"89caee667c866bb1e6b967622514230eff9143cd61b3c41bb16d8f4d8ea878ff"}
Oct 30 22:21:53.484817 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:53.484788    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k" event={"ID":"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7","Type":"ContainerStarted","Data":"cef8f79b554872c3647b8b56eda2e7064a8bd109319a539243facf040b2c124c"}
Oct 30 22:21:53.484992 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:53.484975    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k" event={"ID":"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7","Type":"ContainerStarted","Data":"092efeb1f1d53fbbaaed80f82b05242164ed59b80fa8db85d8b8a189d731a7e5"}
Oct 30 22:21:53.527292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:53.527204    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv" podStartSLOduration=1.52719061 podStartE2EDuration="1.52719061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:53.525998078 +0000 UTC m=+898.882364395" watchObservedRunningTime="2026-10-30 22:21:53.52719061 +0000 UTC m=+898.883556925"
Oct 30 22:21:53.574009 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:53.573943    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k" podStartSLOduration=1.573921258 podStartE2EDuration="1.573921258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:53.57162478 +0000 UTC m=+898.927991118" watchObservedRunningTime="2026-10-30 22:21:53.573921258 +0000 UTC m=+898.930287575"
Oct 30 22:21:53.807619 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:53.807585    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:21:53.826757 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:53.826731    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:21:53.867091 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:53.867066    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:21:53.899400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:53.899354    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:21:54.181294 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:54.176800    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:54.502784 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:54.502759    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:21:54.868870 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:54.868789    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 22:21:54.869384 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:54.869354    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv" podUID="408d956d-ee1c-44e8-9bc5-8d3fc72ab890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 22:21:54.900638 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:54.900612    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 22:21:54.900804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:54.900660    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k" podUID="bd1e5c9d-b6f4-4326-8b97-9f0087c600c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 22:21:54.990660 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:54.990632    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 22:21:54.991931 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:54.991908    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 22:21:55.868119 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:55.868083    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 22:21:55.868324 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:55.868138    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv" podUID="408d956d-ee1c-44e8-9bc5-8d3fc72ab890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 22:21:55.899819 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:55.899786    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 22:21:55.900164 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:55.899836    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k" podUID="bd1e5c9d-b6f4-4326-8b97-9f0087c600c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 22:21:56.441844 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:56.440979    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:56.867607 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:56.867577    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 22:21:56.867796 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:56.867638    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv" podUID="408d956d-ee1c-44e8-9bc5-8d3fc72ab890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 22:21:56.899433 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:56.899353    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 22:21:56.899433 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:56.899398    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k" podUID="bd1e5c9d-b6f4-4326-8b97-9f0087c600c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 22:21:57.867847 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:57.867808    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 22:21:57.868280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:57.867872    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv" podUID="408d956d-ee1c-44e8-9bc5-8d3fc72ab890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 22:21:57.899734 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:57.899707    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 22:21:57.899904 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:57.899762    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k" podUID="bd1e5c9d-b6f4-4326-8b97-9f0087c600c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 22:21:58.551260 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:58.548297    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:58.867840 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:58.867760    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 22:21:58.867840 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:58.867810    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv" podUID="408d956d-ee1c-44e8-9bc5-8d3fc72ab890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 22:21:58.899614 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:58.899587    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 22:21:58.899756 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:58.899633    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k" podUID="bd1e5c9d-b6f4-4326-8b97-9f0087c600c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 22:21:59.868093 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:59.868058    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 22:21:59.868506 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:59.868112    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv" podUID="408d956d-ee1c-44e8-9bc5-8d3fc72ab890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 22:21:59.900150 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:59.900120    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 22:21:59.900338 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:21:59.900180    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k" podUID="bd1e5c9d-b6f4-4326-8b97-9f0087c600c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 22:22:00.441428 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:00.441382    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:00.906155 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:00.885633    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 22:22:00.906155 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:00.885734    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv" podUID="408d956d-ee1c-44e8-9bc5-8d3fc72ab890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 22:22:01.960252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:01.958472    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:22:01.960252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:01.958533    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k" podUID="bd1e5c9d-b6f4-4326-8b97-9f0087c600c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:02.739306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:02.738299    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:22:02.751251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:02.739753    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:22:02.757248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:02.756441    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:22:02.979252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:02.969468    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:03.071340 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:03.051776    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:22:03.071340 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:03.051871    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv" podUID="408d956d-ee1c-44e8-9bc5-8d3fc72ab890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:03.216274 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:03.211769    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:03.510439 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:03.510386    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:03.604283 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:03.582011    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:03.604283 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:03.595400    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:03.677615 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:03.677574    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:03.825310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:03.711566    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:22:03.825310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:03.725038    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:22:03.892253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:03.865621    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:22:03.892253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:03.882538    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:22:03.912251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:03.898741    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:03.960252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:03.938396    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:22:03.988359 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:03.988319    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:04.185015 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:03.989250    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:22:04.185015 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:04.020977    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded"
Oct 30 22:22:04.317289 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:04.208457    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:22:04.317289 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:04.212926    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:04.317289 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:04.232032    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded"
Oct 30 22:22:04.317289 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:04.232885    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:22:04.594419 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:04.594365    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 22:22:04.595534 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:04.595511    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:22:04.633259 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:04.602899    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:04.673632 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:04.605832    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:04.674493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:04.674468    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:22:04.674690 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:04.674674    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:22:04.687446 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:04.664734    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded"
Oct 30 22:22:04.741399 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:04.741315    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": context deadline exceeded"
Oct 30 22:22:04.775390 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:04.775332    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:04.843273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:04.795205    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:22:04.843273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:04.820352    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:04.922914 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:04.895810    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:22:05.000571 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:05.000519    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:05.218880 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:05.218780    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:05.220078 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:05.220044    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:05.233393 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:05.233346    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:05.489317 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:05.485995    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded"
Oct 30 22:22:05.561790 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:05.561751    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:22:05.562126 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:05.562108    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:22:05.562371 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:05.562358    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:22:05.672374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:05.645193    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:05.904556 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:05.904497    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:05.942189 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:05.942141    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:22:05.963970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:05.849592    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:22:06.020086 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:05.849553    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 22:22:06.136720 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:06.058845    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 22:22:06.519255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:06.510402    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded"
Oct 30 22:22:06.563251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:06.562403    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 22:22:06.579439 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:06.578337    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:22:06.594514 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:06.594478    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:22:06.594797 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:06.594767    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t" podUID="32e978e1-1585-4947-8975-fe20d4942bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:06.688311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:06.650328    4889 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:22:06.723360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:06.723305    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg" podUID="7221740b-e003-4af1-86a2-396e32fff7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:06.950311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:06.904990    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:06.950311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:06.905533    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:07.027247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:07.026537    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:07.027247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:07.026721    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:22:07.165154 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:07.140560    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 22:22:07.165154 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:07.140745    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 22:22:07.582455 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:07.568131    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:07.582455 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:07.582347    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:07.622263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:07.617003    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:22:07.746876 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:07.746843    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:07.862347 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:07.858664    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:22:07.950486 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:07.937371    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:08.084286 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:08.058911    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:08.084286 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:08.080673    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:22:08.147382 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:08.147286    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:08.148490 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:08.148466    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:22:08.154240 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:08.149413    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:08.180246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:08.180084    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:22:08.181246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:08.181208    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:22:08.200891 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:08.197533    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:22:08.442301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:08.432105    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:22:08.442301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:08.432198    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:22:08.442301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:08.432279    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:22:10.229787 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:10.229749    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:10.338999 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:22:10.338959    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:22:12.095883 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:12.095826    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:13.774382 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:13.774326    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:15.545403 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:15.545370    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:17.360156 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:17.360117    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:19.142250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:19.141778    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:20.882940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:20.882884    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:22.939250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:22.938322    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:22:22.999910 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:22.999869    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:23.001484 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:23.001330    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 22:22:23.159248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:23.154410    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:23.254255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:23.250020    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:23.315260 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:23.298743    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"]
Oct 30 22:22:23.707479 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:23.621211    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:23.758257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:23.753411    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:23.823288 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:23.818440    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:23.892329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:23.892282    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded"
Oct 30 22:22:23.897596 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:23.897553    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 22:22:23.921250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:23.911800    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"]
Oct 30 22:22:24.084315 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:23.977892    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:24.098795 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.027100    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"]
Oct 30 22:22:24.122328 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.028947    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.151310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.028970    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:24.152939 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.056897    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:24.153442 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.089673    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.153737 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.153713    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.153889 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.153873    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3935b91c-508a-4e8d-99f2-23281921afba-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.154143 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.154121    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.176300 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.176261    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnw2h\" (UniqueName: \"kubernetes.io/projected/3935b91c-508a-4e8d-99f2-23281921afba-kube-api-access-tnw2h\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.176653 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.176629    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.182414 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.182379    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.182690 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.182671    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3935b91c-508a-4e8d-99f2-23281921afba-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.182904 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.182887    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3935b91c-508a-4e8d-99f2-23281921afba-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.192546 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.192502    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 22:22:24.194629 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.194598    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:22:24.290729 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.290674    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:22:24.291063 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.291037    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"]
Oct 30 22:22:24.317102 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.317066    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3935b91c-508a-4e8d-99f2-23281921afba-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.328265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.327387    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3935b91c-508a-4e8d-99f2-23281921afba-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.328265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.327524    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.328265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.327625    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.328265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.327661    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3935b91c-508a-4e8d-99f2-23281921afba-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.328265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.327734    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.328265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.327811    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnw2h\" (UniqueName: \"kubernetes.io/projected/3935b91c-508a-4e8d-99f2-23281921afba-kube-api-access-tnw2h\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.328265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.327912    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.328265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.328009    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.349308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.348080    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3935b91c-508a-4e8d-99f2-23281921afba-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.349308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.348167    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.371637 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.371600    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.393210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.374795    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3935b91c-508a-4e8d-99f2-23281921afba-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.393210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.376380    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:24.393210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.387335    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:22:24.393210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.387442    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:22:24.393210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.388218    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 22:22:24.393210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.389022    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:22:24.398798 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.398736    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3935b91c-508a-4e8d-99f2-23281921afba-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.411714 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.411679    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.416581 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.416506    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.417019 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.416997    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.464530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.429829    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:24.464530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.429939    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:24.464530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.430060    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3084be58-a1e9-410a-b501-2c71cca33742-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:24.464530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.430144    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:24.464530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.430284    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:24.464530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.430328    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3084be58-a1e9-410a-b501-2c71cca33742-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:24.464530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.430366    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjfhn\" (UniqueName: \"kubernetes.io/projected/3084be58-a1e9-410a-b501-2c71cca33742-kube-api-access-fjfhn\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:24.464530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.430456    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3084be58-a1e9-410a-b501-2c71cca33742-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:24.464530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.430531    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:24.648313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.629096    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnw2h\" (UniqueName: \"kubernetes.io/projected/3935b91c-508a-4e8d-99f2-23281921afba-kube-api-access-tnw2h\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.656315 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.651453    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:22:24.688950 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.688912    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:24.689295 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.689271    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:24.689549 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.689527    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3084be58-a1e9-410a-b501-2c71cca33742-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:24.689736 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.689718    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:24.689951 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.689934    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:24.690091 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.690073    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3084be58-a1e9-410a-b501-2c71cca33742-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:24.690212 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.690195    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjfhn\" (UniqueName: \"kubernetes.io/projected/3084be58-a1e9-410a-b501-2c71cca33742-kube-api-access-fjfhn\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:24.690496 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.690476    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3084be58-a1e9-410a-b501-2c71cca33742-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:24.690680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.690664    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:24.754341 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.753750    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:24.754546 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.754442    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3084be58-a1e9-410a-b501-2c71cca33742-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:24.761256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.754658    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:24.761256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.754884    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:24.822416 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.820617    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded"
Oct 30 22:22:24.822416 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.821199    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:24.875923 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.822310    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": context deadline exceeded"
Oct 30 22:22:24.949457 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.949341    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:24.994477 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.994442    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:24.999247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:24.995603    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded"
Oct 30 22:22:25.011895 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:25.001434    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded"
Oct 30 22:22:25.031363 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:25.031320    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:22:25.114705 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:25.114633    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3084be58-a1e9-410a-b501-2c71cca33742-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:25.201309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:25.171215    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:25.394251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:25.189770    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:22:25.394883 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:25.202384    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjfhn\" (UniqueName: \"kubernetes.io/projected/3084be58-a1e9-410a-b501-2c71cca33742-kube-api-access-fjfhn\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:25.395073 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:25.227399    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:25.395437 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:25.228718    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:22:25.395659 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:25.395641    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:22:25.395806 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:25.395794    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:22:25.395941 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:25.395930    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:22:25.396093 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:25.396081    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:22:25.396565 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:25.271468    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:25.490311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:25.474445    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:25.608288 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:25.600995    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3084be58-a1e9-410a-b501-2c71cca33742-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:25.924588 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:25.924538    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:25.957989 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:25.957949    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:26.084550 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:25.999554    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:26.085105 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:26.047665    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:22:26.085524 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:26.077681    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97" podUID="b491f9ed-944c-48f8-95db-a907d8bfc92c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:26.085947 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:26.085925    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:22:26.125132 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:26.125080    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:26.423135 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:26.421905    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:26.423135 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:26.422292    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 22:22:26.423135 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:26.422557    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:22:26.484380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:26.484337    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:22:26.539777 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:26.539740    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:26.595251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:26.570198    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:27.012630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:26.974447    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:27.102455 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:27.102395    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded"
Oct 30 22:22:27.324701 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:27.121651    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 22:22:27.352284 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:27.170214    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:22:27.368392 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:27.229675    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:27.368392 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:27.362984    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:22:27.368392 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:27.365276    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:22:27.426243 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:27.426166    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:27.449864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:27.426392    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 22:22:27.456301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:27.426444    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 22:22:27.500561 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:27.500517    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:28.075261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:28.055405    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:22:28.531519 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:28.525875    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:28.561399 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:28.561343    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:28.632450 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:28.575996    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 22:22:28.633064 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:28.632934    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:22:29.108344 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:28.633275    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:29.108344 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:28.633296    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 22:22:29.108344 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:28.633397    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:22:29.108344 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:28.815629    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:29.125804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:29.125765    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pxdsm" podUID="22e10730-dd08-4c76-8b7c-7b87e23e571d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:29.189446 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:29.161189    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97" podUID="b491f9ed-944c-48f8-95db-a907d8bfc92c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": context deadline exceeded"
Oct 30 22:22:29.523472 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:29.523210    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:29.635301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:29.635158    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 22:22:29.677397 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:29.677348    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:22:29.683592 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:29.683565    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:22:29.704426 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:29.704401    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:22:29.705904 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:29.705884    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:22:29.707588 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:29.707506    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:22:29.712108 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:29.712093    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:22:29.714466 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:29.714449    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:22:29.714670 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:29.714656    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:22:29.715693 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:29.692263    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:22:29.716777 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:29.694318    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:22:29.716777 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:29.716460    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:22:29.738629 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:29.738589    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:22:29.741003 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:29.740799    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:22:29.744298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:29.744272    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:22:29.746017 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:29.745985    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:22:29.765601 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:29.765576    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:22:29.765746 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:29.765712    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"]
Oct 30 22:22:29.770929 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:29.770893    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"]
Oct 30 22:22:30.674401 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:30.671346    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2" event={"ID":"3935b91c-508a-4e8d-99f2-23281921afba","Type":"ContainerStarted","Data":"611eb33d36595091c65fa3a504b77762a888833dd3fb07c2b191e0d2658937f4"}
Oct 30 22:22:30.674401 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:30.671406    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2" event={"ID":"3935b91c-508a-4e8d-99f2-23281921afba","Type":"ContainerStarted","Data":"083c3104495c24f5d0dfe435fc6de6e88d7ef3090f02bd97df0fd7627644d775"}
Oct 30 22:22:30.674401 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:30.673372    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt" event={"ID":"3084be58-a1e9-410a-b501-2c71cca33742","Type":"ContainerStarted","Data":"0acf6cf2a7c17874651cd2ea4bdb41059fc147d1ecf885d2b013ff68be84a870"}
Oct 30 22:22:30.674401 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:30.673410    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt" event={"ID":"3084be58-a1e9-410a-b501-2c71cca33742","Type":"ContainerStarted","Data":"0873359859d82861b7a9fe0680d480e8cb5b502f6efb2fc872234663f64538a4"}
Oct 30 22:22:30.802245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:30.802167    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2" podStartSLOduration=7.802152527 podStartE2EDuration="7.802152527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:30.757585617 +0000 UTC m=+936.113951931" watchObservedRunningTime="2026-10-30 22:22:30.802152527 +0000 UTC m=+936.158518841"
Oct 30 22:22:30.804403 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:30.804359    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt" podStartSLOduration=7.80434689 podStartE2EDuration="7.80434689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:30.800403005 +0000 UTC m=+936.156769319" watchObservedRunningTime="2026-10-30 22:22:30.80434689 +0000 UTC m=+936.160713267"
Oct 30 22:22:30.971095 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:30.971013    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:30.972135 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:30.972111    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 22:22:30.972188 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:30.972160    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2" podUID="3935b91c-508a-4e8d-99f2-23281921afba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 22:22:31.617249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:31.615269    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:31.617249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:31.615389    4889 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:22:31.617249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:31.615431    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt" podUID="3084be58-a1e9-410a-b501-2c71cca33742" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 22:22:31.706255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:31.698169    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:31.971280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:31.971170    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 22:22:31.971280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:31.971254    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2" podUID="3935b91c-508a-4e8d-99f2-23281921afba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 22:22:32.630253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:32.625341    4889 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:22:32.630253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:32.625398    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt" podUID="3084be58-a1e9-410a-b501-2c71cca33742" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 22:22:32.971788 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:32.971697    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 22:22:32.971788 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:32.971763    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2" podUID="3935b91c-508a-4e8d-99f2-23281921afba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 22:22:33.618683 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:33.618645    4889 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:22:33.618943 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:33.618915    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt" podUID="3084be58-a1e9-410a-b501-2c71cca33742" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 22:22:33.825879 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:33.825842    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:33.976876 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:33.976794    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 22:22:33.977471 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:33.977442    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2" podUID="3935b91c-508a-4e8d-99f2-23281921afba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 22:22:34.634249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:34.633652    4889 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:22:34.634249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:34.633713    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt" podUID="3084be58-a1e9-410a-b501-2c71cca33742" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 22:22:34.972050 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:34.971939    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 22:22:34.972278 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:34.972257    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2" podUID="3935b91c-508a-4e8d-99f2-23281921afba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 22:22:35.632691 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:35.632641    4889 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:22:35.633326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:35.633291    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt" podUID="3084be58-a1e9-410a-b501-2c71cca33742" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 22:22:35.943869 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:35.943783    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:35.971421 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:35.971386    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 22:22:35.971693 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:35.971667    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2" podUID="3935b91c-508a-4e8d-99f2-23281921afba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 22:22:36.612041 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:36.612003    4889 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:22:36.612260 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:36.612059    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt" podUID="3084be58-a1e9-410a-b501-2c71cca33742" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 22:22:36.971660 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:36.971583    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 22:22:36.971660 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:36.971640    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2" podUID="3935b91c-508a-4e8d-99f2-23281921afba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 22:22:37.615832 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:37.615803    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:37.616266 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:37.616251    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:37.616983 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:37.616963    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:22:37.807940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:37.807897    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:38.007109 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.007063    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"]
Oct 30 22:22:38.024617 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.024582    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:38.031490 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.028135    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"]
Oct 30 22:22:38.031883 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.031845    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.037460 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.037394    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:38.078438 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.078400    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:22:38.115894 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.115861    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"]
Oct 30 22:22:38.132727 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.132699    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.146313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.146282    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.146621 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.146597    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.148004 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.147981    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.148853 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.148566    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.149034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.149015    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.149887 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.149845    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.150156 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.150129    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.150621 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.150603    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.151006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.150988    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqdb9\" (UniqueName: \"kubernetes.io/projected/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-kube-api-access-jqdb9\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.152977 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.152959    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"]
Oct 30 22:22:38.253168 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.253135    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.253413 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.253179    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.253413 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.253202    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.253413 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.253264    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.253413 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.253283    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4qff\" (UniqueName: \"kubernetes.io/projected/d15e2f68-d995-43a9-aa5e-32aec47af6f7-kube-api-access-z4qff\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.253413 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.253319    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.253413 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.253338    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqdb9\" (UniqueName: \"kubernetes.io/projected/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-kube-api-access-jqdb9\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.253413 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.253365    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.253413 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.253393    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.253413 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.253420    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.253867 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.253448    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.253867 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.253471    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.253867 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.253496    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.253867 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.253519    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.253867 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.253551    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.253867 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.253624    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.253867 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.253643    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.253867 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.253684    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.254865 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.254844    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.255050 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.254889    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.255924 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.255899    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.256169 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.256144    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.256425 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.256408    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.259255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.259176    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.259362 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.259342    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.275256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.275211    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.275405 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.275319    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqdb9\" (UniqueName: \"kubernetes.io/projected/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-kube-api-access-jqdb9\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.354542 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.354506    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.354760 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.354571    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.354760 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.354596    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.354872 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.354797    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.354872 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.354849    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.354965 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.354919    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4qff\" (UniqueName: \"kubernetes.io/projected/d15e2f68-d995-43a9-aa5e-32aec47af6f7-kube-api-access-z4qff\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.354965 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.354926    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.354965 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.354951    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.355065 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.354999    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.355065 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.355007    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.355125 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.355105    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.355158 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.355144    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.355503 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.355476    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.355776 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.355605    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.357376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.357345    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.357611 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.357583    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.375760 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.375731    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.375866 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.375853    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4qff\" (UniqueName: \"kubernetes.io/projected/d15e2f68-d995-43a9-aa5e-32aec47af6f7-kube-api-access-z4qff\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.466628 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.466599    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:38.474002 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.473977    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:38.571169 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.571137    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"]
Oct 30 22:22:38.579033 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.579004    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.592624 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.592589    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"]
Oct 30 22:22:38.668020 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.667981    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-data\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.668183 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.668045    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-envoy\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.668183 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.668148    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-credential-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.668290 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.668187    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcskg\" (UniqueName: \"kubernetes.io/projected/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-kube-api-access-vcskg\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.668471 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.668447    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istiod-ca-cert\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.668538 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.668496    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-workload-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.668606 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.668589    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-workload-certs\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.668740 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.668723    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-podinfo\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.668772 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.668756    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-token\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.770112 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.770075    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istiod-ca-cert\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.770112 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.770130    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-workload-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.770449 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.770156    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-workload-certs\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.770449 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.770278    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-podinfo\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.770449 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.770300    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-token\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.770449 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.770329    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-data\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.770449 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.770384    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-envoy\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.770449 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.770435    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-credential-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.771367 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.770471    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcskg\" (UniqueName: \"kubernetes.io/projected/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-kube-api-access-vcskg\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.771367 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.770596    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-workload-certs\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.771367 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.770874    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istiod-ca-cert\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.771367 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.770885    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-workload-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.771367 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.770942    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-credential-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.771367 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.771300    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-data\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.773866 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.773783    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-envoy\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.778473 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.778447    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-podinfo\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.787617 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:22:38.787564    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0770be9_8ba4_4b85_8c96_6e6dab7a0ba5.slice/crio-c6d0090bdf599acba2d28d834982bf7525754780395f2cc78dde03e1375e97da WatchSource:0}: Error finding container c6d0090bdf599acba2d28d834982bf7525754780395f2cc78dde03e1375e97da: Status 404 returned error can't find the container with id c6d0090bdf599acba2d28d834982bf7525754780395f2cc78dde03e1375e97da
Oct 30 22:22:38.789586 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.789545    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"]
Oct 30 22:22:38.790418 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:22:38.790396    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd15e2f68_d995_43a9_aa5e_32aec47af6f7.slice/crio-2f7cc7d3a64200486cd54b086f307e48032d05ee69f09339491b0512f2f1a1b0 WatchSource:0}: Error finding container 2f7cc7d3a64200486cd54b086f307e48032d05ee69f09339491b0512f2f1a1b0: Status 404 returned error can't find the container with id 2f7cc7d3a64200486cd54b086f307e48032d05ee69f09339491b0512f2f1a1b0
Oct 30 22:22:38.790552 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.790517    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:22:38.790616 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.790603    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:22:38.790672 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.790649    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:22:38.791575 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.791343    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-token\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.791575 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.791427    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcskg\" (UniqueName: \"kubernetes.io/projected/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-kube-api-access-vcskg\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:38.794047 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.794018    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:22:38.794113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.794096    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:22:38.794160 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.794116    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"]
Oct 30 22:22:38.794160 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.794146    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:22:38.805474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.805444    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w" event={"ID":"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5","Type":"ContainerStarted","Data":"c6d0090bdf599acba2d28d834982bf7525754780395f2cc78dde03e1375e97da"}
Oct 30 22:22:38.807019 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.806998    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7" event={"ID":"d15e2f68-d995-43a9-aa5e-32aec47af6f7","Type":"ContainerStarted","Data":"2f7cc7d3a64200486cd54b086f307e48032d05ee69f09339491b0512f2f1a1b0"}
Oct 30 22:22:38.920038 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:38.920004    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:39.259620 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:22:39.259588    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ca58fb_2ebf_452a_a6d5_26311c091b1c.slice/crio-ccabc8b6ef653852d43ce5a251f6d6723d6348805ee0cc7182b290358a3585be WatchSource:0}: Error finding container ccabc8b6ef653852d43ce5a251f6d6723d6348805ee0cc7182b290358a3585be: Status 404 returned error can't find the container with id ccabc8b6ef653852d43ce5a251f6d6723d6348805ee0cc7182b290358a3585be
Oct 30 22:22:39.260733 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:39.260704    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"]
Oct 30 22:22:39.263083 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:39.263041    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:22:39.263199 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:39.263130    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:22:39.263199 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:39.263173    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:22:39.839254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:39.836263    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:39.996305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:39.958867    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w" event={"ID":"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5","Type":"ContainerStarted","Data":"4b1f9d8a2197c754b46f0d96fe1295bc6f9040b6db73bd6ae1038e5a3d4ba651"}
Oct 30 22:22:40.003885 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:40.003848    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr" event={"ID":"b6ca58fb-2ebf-452a-a6d5-26311c091b1c","Type":"ContainerStarted","Data":"7cd10e61e23312ecc757569cf81ae5eece647767404f2ec36ebccf667df99436"}
Oct 30 22:22:40.004180 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:40.004151    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr" event={"ID":"b6ca58fb-2ebf-452a-a6d5-26311c091b1c","Type":"ContainerStarted","Data":"ccabc8b6ef653852d43ce5a251f6d6723d6348805ee0cc7182b290358a3585be"}
Oct 30 22:22:40.179571 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:40.179352    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w" podStartSLOduration=3.179323303 podStartE2EDuration="3.179323303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:40.058006528 +0000 UTC m=+945.414372845" watchObservedRunningTime="2026-10-30 22:22:40.179323303 +0000 UTC m=+945.535689614"
Oct 30 22:22:40.186191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:40.186156    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7" event={"ID":"d15e2f68-d995-43a9-aa5e-32aec47af6f7","Type":"ContainerStarted","Data":"6aeac9d62246834e8a096dc043515debab44db6b8421e554ea1e31ef13f60a54"}
Oct 30 22:22:40.199081 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:40.199005    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr" podStartSLOduration=2.198990024 podStartE2EDuration="2.198990024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:40.137958526 +0000 UTC m=+945.494324842" watchObservedRunningTime="2026-10-30 22:22:40.198990024 +0000 UTC m=+945.555356337"
Oct 30 22:22:40.271427 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:40.271362    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7" podStartSLOduration=2.2713415 podStartE2EDuration="2.2713415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:40.24398598 +0000 UTC m=+945.600352296" watchObservedRunningTime="2026-10-30 22:22:40.2713415 +0000 UTC m=+945.627707815"
Oct 30 22:22:40.363545 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:40.363508    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:40.389118 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:22:40.385959    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:22:40.472311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:40.471417    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:40.477830 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:40.476324    4889 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:22:40.477830 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:40.476384    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w" podUID="b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 22:22:40.477830 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:40.476514    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:40.478334 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:40.478202    4889 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 22:22:40.478334 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:40.478274    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7" podUID="d15e2f68-d995-43a9-aa5e-32aec47af6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 22:22:40.608251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:40.607464    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:40.922251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:40.921605    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:40.926249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:40.924264    4889 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 22:22:40.926249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:40.924318    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr" podUID="b6ca58fb-2ebf-452a-a6d5-26311c091b1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 22:22:41.489571 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:41.487600    4889 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 22:22:41.489571 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:41.487665    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7" podUID="d15e2f68-d995-43a9-aa5e-32aec47af6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 22:22:41.496255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:41.495773    4889 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:22:41.496255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:41.495860    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w" podUID="b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 22:22:41.942248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:41.941577    4889 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 22:22:41.942248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:41.941656    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr" podUID="b6ca58fb-2ebf-452a-a6d5-26311c091b1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 22:22:42.199302 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:42.182670    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:42.467975 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:42.467886    4889 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:22:42.468204 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:42.468176    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w" podUID="b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 22:22:42.477150 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:42.477119    4889 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 22:22:42.477410 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:42.477380    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7" podUID="d15e2f68-d995-43a9-aa5e-32aec47af6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 22:22:42.934688 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:42.934652    4889 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 22:22:42.935319 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:42.935288    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr" podUID="b6ca58fb-2ebf-452a-a6d5-26311c091b1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 22:22:43.468717 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:43.468665    4889 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:22:43.470203 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:43.470168    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w" podUID="b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 22:22:43.476763 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:43.476733    4889 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 22:22:43.476956 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:43.476931    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7" podUID="d15e2f68-d995-43a9-aa5e-32aec47af6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 22:22:43.928651 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:43.928612    4889 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 22:22:43.929022 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:43.928979    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr" podUID="b6ca58fb-2ebf-452a-a6d5-26311c091b1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 22:22:44.403249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:44.402822    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:44.468812 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:44.468779    4889 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:22:44.469084 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:44.469050    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w" podUID="b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 22:22:44.485255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:44.474821    4889 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 22:22:44.485566 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:44.485527    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7" podUID="d15e2f68-d995-43a9-aa5e-32aec47af6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 22:22:44.971262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:44.970902    4889 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 22:22:44.971262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:44.970975    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr" podUID="b6ca58fb-2ebf-452a-a6d5-26311c091b1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 22:22:45.467103 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:45.467066    4889 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:22:45.467585 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:45.467560    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w" podUID="b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 22:22:45.475136 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:45.475103    4889 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 22:22:45.475292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:45.475152    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7" podUID="d15e2f68-d995-43a9-aa5e-32aec47af6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 22:22:45.924634 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:45.924598    4889 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 22:22:45.924929 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:45.924899    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr" podUID="b6ca58fb-2ebf-452a-a6d5-26311c091b1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 22:22:46.146880 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:46.146839    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:46.467069 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:46.467035    4889 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:22:46.467363 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:46.467335    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w" podUID="b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 22:22:46.602254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:46.596172    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:46.920467 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:46.920425    4889 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 22:22:46.920650 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:46.920488    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr" podUID="b6ca58fb-2ebf-452a-a6d5-26311c091b1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 22:22:47.245305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:47.237607    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:47.245305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:47.238883    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:47.245305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:47.240458    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:22:47.487985 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:47.487946    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:47.489438 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:47.489417    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:47.489901 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:47.489884    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:22:47.925263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:47.925203    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:47.925750 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:47.925731    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:47.926393 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:47.926378    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:22:48.520363 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:48.520329    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:50.494795 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:50.492177    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:52.338251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:52.338007    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:54.277900 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:54.277860    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:56.137362 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:56.135470    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:57.945002 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:22:57.944823    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:00.151251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:00.130081    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:00.176251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:00.175389    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:00.223300 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:00.222333    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:00.312252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:00.311979    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:00.377670 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:00.377587    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:00.436311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:00.424364    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:00.436311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:00.433767    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 22:23:00.500291 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:00.495326    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:00.556418 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:00.536424    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded"
Oct 30 22:23:01.152251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:01.144392    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 22:23:01.152251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:01.145310    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:23:01.203575 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:01.203515    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:01.234251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:01.209006    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:23:01.287499 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:01.250321    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:01.288069 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:01.261359    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:01.292327 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:01.292288    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw" podUID="d441e1d3-9571-4513-8404-2d6569566ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:01.292951 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:01.292916    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:23:01.590852 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:01.587872    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:01.592741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:01.592701    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:01.650663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:01.650607    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded"
Oct 30 22:23:01.708301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:01.661077    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:23:01.708301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:01.661356    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:23:01.713273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:01.710060    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 22:23:01.925019 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:01.717823    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:23:02.124341 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:02.124310    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:23:02.159202 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:02.159168    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:23:02.307799 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:02.307743    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:23:02.363391 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:02.363344    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:02.364639 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:02.364604    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:02.394427 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:02.394366    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:23:02.472250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:02.408921    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:02.539250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:02.515711    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:23:02.671318 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:02.650568    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:02.671318 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:02.650632    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh" podUID="89d5a590-0ac8-4be8-a791-942b739fc869" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:02.671318 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:02.654301    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:02.671318 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:02.655109    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 22:23:03.169254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:03.144398    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 22:23:03.169254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:03.144808    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:23:03.324593 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:03.324201    4889 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:03.324593 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:03.324290    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6" podUID="c2a808e4-e5d8-415e-960e-1b21c1368d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.357252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:03.326817    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.379363 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:03.379325    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:03.379628 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:03.379602    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh" podUID="69ae6469-721f-4879-bd5f-132fbc422276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.540249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:03.539387    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 22:23:03.540249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:03.539453    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:23:03.540581 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:03.540274    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.540581 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:03.540576    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:23:03.569258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:03.541005    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.638257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:03.583891    4889 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:03.638257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:03.583969    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w" podUID="b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.638257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:03.584081    4889 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:03.638257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:03.584107    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7" podUID="d15e2f68-d995-43a9-aa5e-32aec47af6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.638257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:03.584184    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:03.638257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:03.584210    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8" podUID="f08b855f-771b-4002-9555-c7d56ba510c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.670251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:03.655299    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.670251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:03.654521    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.670251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:03.656872    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.819844 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:03.819707    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.898352 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:03.896817    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:04.163310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:04.146036    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:04.163310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:04.146536    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:04.340986 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:04.340375    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:23:04.552044 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:04.549790    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:04.552044 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:04.550178    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:04.552044 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:04.550465    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:23:04.568007 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:04.565861    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:04.653380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:04.652946    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:04.672307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:04.672262    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded"
Oct 30 22:23:04.673124 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:04.672745    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:04.742755 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:04.742702    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": context deadline exceeded"
Oct 30 22:23:04.850311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:04.825423    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:04.948300 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:04.936742    4889 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:05.039310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:04.959781    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv" podUID="408d956d-ee1c-44e8-9bc5-8d3fc72ab890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:05.146854 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:05.146730    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:05.147990 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:05.147960    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:23:05.342483 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:05.342378    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:23:05.608252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:05.529528    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:05.608252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:05.550324    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:05.608252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:05.551199    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:05.608252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:05.551504    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 22:23:05.608252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:05.591334    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:05.686265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:05.686185    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:05.718252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:05.687982    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:06.188249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:06.184971    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:06.197253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:06.189903    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:23:06.252952 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:06.252916    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:06.253254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:06.253197    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t" podUID="32e978e1-1585-4947-8975-fe20d4942bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:06.313810 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:06.313772    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:06.357369 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:06.356409    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:06.579253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:06.558339    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:23:06.579253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:06.563313    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:23:06.616249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:06.581436    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 22:23:06.617023 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:06.616985    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:06.738422 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:06.738383    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:06.782584 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:06.738659    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 22:23:07.189756 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:07.189714    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:07.211429 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:07.210281    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:23:07.376122 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:07.376079    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:07.562249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:07.559922    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:07.572297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:07.569804    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:07.609249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:07.596825    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 22:23:07.620605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:07.620566    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 22:23:07.819480 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:07.819376    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:07.823296 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:07.823072    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:08.080303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:08.076456    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:23:08.089011 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:08.088152    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:23:08.091077 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:08.091011    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:23:08.092782 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:08.092768    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:23:08.098441 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:08.098426    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:23:08.101193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:08.101177    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:23:08.189785 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:08.189755    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:23:08.240250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:08.239895    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:23:08.295034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:08.294994    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:23:09.470248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:09.468909    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:10.302349 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:23:10.302316    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:23:11.223116 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:11.223080    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:14.004894 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:14.004862    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:14.638315 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:14.638270    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:15.926825 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:15.926774    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:16.442249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:16.441650    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:18.177529 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:18.177420    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:18.915441 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:18.915410    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:19.347692 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:19.347635    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:19.515444 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:19.515411    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:19.698963 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:19.698785    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:19.939453 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:19.939422    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:20.116880 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:20.116829    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:20.343924 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:20.343749    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:20.568932 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:20.568889    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:20.699102 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:20.699069    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"]
Oct 30 22:23:20.706415 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:20.706391    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:20.755876 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:20.755848    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-36d7ee39-openshift-default-dockercfg-4zc68\""
Oct 30 22:23:20.782756 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:20.782561    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:20.843317 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:20.809733    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-token\") pod \"e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") " pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.010350 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:20.982872    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-credential-socket\") pod \"e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") " pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.010350 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:20.985061    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vzrl\" (UniqueName: \"kubernetes.io/projected/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-kube-api-access-8vzrl\") pod \"e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") " pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.010350 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:20.985191    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-workload-socket\") pod \"e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") " pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.010350 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:20.985268    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-podinfo\") pod \"e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") " pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.010350 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:20.985327    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istiod-ca-cert\") pod \"e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") " pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.010350 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:20.985460    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-workload-certs\") pod \"e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") " pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.010350 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:20.985593    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-data\") pod \"e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") " pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.010350 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:20.985654    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-envoy\") pod \"e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") " pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.010350 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:20.986874    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:21.074452 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:21.067187    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"]
Oct 30 22:23:21.098769 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:21.098537    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istiod-ca-cert\") pod \"e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") " pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.099050 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:21.099030    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-workload-certs\") pod \"e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") " pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.099250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:21.099215    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-data\") pod \"e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") " pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.099364 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:21.099352    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-envoy\") pod \"e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") " pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.099465 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:21.099454    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-token\") pod \"e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") " pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.099620 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:21.099609    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-credential-socket\") pod \"e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") " pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.099802 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:21.099787    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vzrl\" (UniqueName: \"kubernetes.io/projected/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-kube-api-access-8vzrl\") pod \"e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") " pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.099946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:21.099907    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istiod-ca-cert\") pod \"e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") " pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.101323 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:21.099965    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-workload-socket\") pod \"e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") " pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.101603 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:21.100275    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-workload-socket\") pod \"e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") " pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.101724 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:21.100393    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-credential-socket\") pod \"e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") " pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.101833 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:21.100793    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-workload-certs\") pod \"e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") " pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.101938 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:21.100926    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-data\") pod \"e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") " pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.102682 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:21.102658    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-podinfo\") pod \"e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") " pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.104992 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:21.104972    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-envoy\") pod \"e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") " pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.105953 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:21.105931    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-podinfo\") pod \"e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") " pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.121027 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:21.121000    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-token\") pod \"e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") " pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.121172 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:21.121027    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vzrl\" (UniqueName: \"kubernetes.io/projected/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-kube-api-access-8vzrl\") pod \"e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") " pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.312799 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:21.311528    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:21.729040 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:23:21.729013    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b1a3f3_f888_408b_b6ac_548b3eef0ca9.slice/crio-3a78d8e4540bb54373e08039be067fe23f6b86cca63c4ef2cb4835aec75eebbc WatchSource:0}: Error finding container 3a78d8e4540bb54373e08039be067fe23f6b86cca63c4ef2cb4835aec75eebbc: Status 404 returned error can't find the container with id 3a78d8e4540bb54373e08039be067fe23f6b86cca63c4ef2cb4835aec75eebbc
Oct 30 22:23:21.730527 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:21.730499    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"]
Oct 30 22:23:21.732092 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:21.732056    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:23:21.732190 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:21.732132    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:23:21.732190 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:21.732176    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:23:22.561843 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:22.561038    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m" event={"ID":"50b1a3f3-f888-408b-b6ac-548b3eef0ca9","Type":"ContainerStarted","Data":"0f4b43efc19244f8b918488e9dc25979f2fe451315913cb960ce8761639f25a6"}
Oct 30 22:23:22.561843 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:22.561095    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m" event={"ID":"50b1a3f3-f888-408b-b6ac-548b3eef0ca9","Type":"ContainerStarted","Data":"3a78d8e4540bb54373e08039be067fe23f6b86cca63c4ef2cb4835aec75eebbc"}
Oct 30 22:23:22.619843 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:22.619796    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m" podStartSLOduration=2.61978165 podStartE2EDuration="2.61978165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:22.616628247 +0000 UTC m=+987.972994562" watchObservedRunningTime="2026-10-30 22:23:22.61978165 +0000 UTC m=+987.976147963"
Oct 30 22:23:22.821915 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:22.821819    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:23.064246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:23.059154    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:23.319923 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:23.319759    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:23.361397 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:23.361345    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:23.567402 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:23.567374    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:23.568426 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:23.568406    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:23:23.842389 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:23.842352    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:24.028720 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:24.028689    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:24.525707 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:24.525673    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:24.702637 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:24.702605    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:24.907336 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:24.905305    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:25.088291 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:25.086047    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:26.496170 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:26.496135    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:26.750588 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:26.750505    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:26.976569 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:26.976538    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:27.224306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:27.217762    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:27.518274 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:27.517838    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn"]
Oct 30 22:23:27.552246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:27.543096    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn"
Oct 30 22:23:27.696187 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:27.672597    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:27.696187 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:27.683351    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn"]
Oct 30 22:23:27.713139 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:27.713113    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-36d7ee39\""
Oct 30 22:23:27.713340 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:27.713212    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-36d7ee39-dockercfg-lb9v6\""
Oct 30 22:23:27.731162 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:27.731123    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4378f9b0-0538-4d64-8697-9bfc2ac57027-plugins-config-volume\") pod \"payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn\" (UID: \"4378f9b0-0538-4d64-8697-9bfc2ac57027\") " pod="openshift-ingress/payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn"
Oct 30 22:23:27.731621 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:27.731599    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb4m6\" (UniqueName: \"kubernetes.io/projected/4378f9b0-0538-4d64-8697-9bfc2ac57027-kube-api-access-fb4m6\") pod \"payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn\" (UID: \"4378f9b0-0538-4d64-8697-9bfc2ac57027\") " pod="openshift-ingress/payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn"
Oct 30 22:23:27.834348 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:27.834258    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb4m6\" (UniqueName: \"kubernetes.io/projected/4378f9b0-0538-4d64-8697-9bfc2ac57027-kube-api-access-fb4m6\") pod \"payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn\" (UID: \"4378f9b0-0538-4d64-8697-9bfc2ac57027\") " pod="openshift-ingress/payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn"
Oct 30 22:23:27.835009 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:27.834963    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4378f9b0-0538-4d64-8697-9bfc2ac57027-plugins-config-volume\") pod \"payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn\" (UID: \"4378f9b0-0538-4d64-8697-9bfc2ac57027\") " pod="openshift-ingress/payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn"
Oct 30 22:23:27.836036 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:27.836010    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4378f9b0-0538-4d64-8697-9bfc2ac57027-plugins-config-volume\") pod \"payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn\" (UID: \"4378f9b0-0538-4d64-8697-9bfc2ac57027\") " pod="openshift-ingress/payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn"
Oct 30 22:23:27.854127 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:27.854104    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb4m6\" (UniqueName: \"kubernetes.io/projected/4378f9b0-0538-4d64-8697-9bfc2ac57027-kube-api-access-fb4m6\") pod \"payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn\" (UID: \"4378f9b0-0538-4d64-8697-9bfc2ac57027\") " pod="openshift-ingress/payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn"
Oct 30 22:23:27.869872 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:27.869836    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:27.955627 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:27.955599    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29"]
Oct 30 22:23:27.959964 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:27.959949    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29"
Oct 30 22:23:27.976074 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:27.976049    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1ba9ae08-dockercfg-hx9km\""
Oct 30 22:23:27.976757 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:27.976735    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1ba9ae08\""
Oct 30 22:23:27.977660 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:27.977638    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29"]
Oct 30 22:23:28.002660 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:28.002629    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48"]
Oct 30 22:23:28.008186 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:28.008166    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48"
Oct 30 22:23:28.012923 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:28.012906    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn"
Oct 30 22:23:28.025698 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:28.025674    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48"]
Oct 30 22:23:28.039388 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:28.039342    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjbf2\" (UniqueName: \"kubernetes.io/projected/cab0b42f-5ee8-4e8b-a6c7-75de81188465-kube-api-access-cjbf2\") pod \"payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29\" (UID: \"cab0b42f-5ee8-4e8b-a6c7-75de81188465\") " pod="openshift-ingress/payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29"
Oct 30 22:23:28.039509 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:28.039448    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/512d1cfc-681e-4360-aff3-791e3f3f854e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48\" (UID: \"512d1cfc-681e-4360-aff3-791e3f3f854e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48"
Oct 30 22:23:28.040447 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:28.040419    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab0b42f-5ee8-4e8b-a6c7-75de81188465-plugins-config-volume\") pod \"payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29\" (UID: \"cab0b42f-5ee8-4e8b-a6c7-75de81188465\") " pod="openshift-ingress/payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29"
Oct 30 22:23:28.040623 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:28.040600    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v77nk\" (UniqueName: \"kubernetes.io/projected/512d1cfc-681e-4360-aff3-791e3f3f854e-kube-api-access-v77nk\") pod \"payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48\" (UID: \"512d1cfc-681e-4360-aff3-791e3f3f854e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48"
Oct 30 22:23:28.065973 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:28.057653    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:28.160360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:28.154096    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjbf2\" (UniqueName: \"kubernetes.io/projected/cab0b42f-5ee8-4e8b-a6c7-75de81188465-kube-api-access-cjbf2\") pod \"payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29\" (UID: \"cab0b42f-5ee8-4e8b-a6c7-75de81188465\") " pod="openshift-ingress/payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29"
Oct 30 22:23:28.160360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:28.154589    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/512d1cfc-681e-4360-aff3-791e3f3f854e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48\" (UID: \"512d1cfc-681e-4360-aff3-791e3f3f854e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48"
Oct 30 22:23:28.160360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:28.154791    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab0b42f-5ee8-4e8b-a6c7-75de81188465-plugins-config-volume\") pod \"payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29\" (UID: \"cab0b42f-5ee8-4e8b-a6c7-75de81188465\") " pod="openshift-ingress/payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29"
Oct 30 22:23:28.160360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:28.155067    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v77nk\" (UniqueName: \"kubernetes.io/projected/512d1cfc-681e-4360-aff3-791e3f3f854e-kube-api-access-v77nk\") pod \"payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48\" (UID: \"512d1cfc-681e-4360-aff3-791e3f3f854e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48"
Oct 30 22:23:28.160360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:28.156537    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/512d1cfc-681e-4360-aff3-791e3f3f854e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48\" (UID: \"512d1cfc-681e-4360-aff3-791e3f3f854e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48"
Oct 30 22:23:28.164264 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:28.163396    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab0b42f-5ee8-4e8b-a6c7-75de81188465-plugins-config-volume\") pod \"payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29\" (UID: \"cab0b42f-5ee8-4e8b-a6c7-75de81188465\") " pod="openshift-ingress/payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29"
Oct 30 22:23:28.324833 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:28.285584    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:28.324833 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:28.304116    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v77nk\" (UniqueName: \"kubernetes.io/projected/512d1cfc-681e-4360-aff3-791e3f3f854e-kube-api-access-v77nk\") pod \"payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48\" (UID: \"512d1cfc-681e-4360-aff3-791e3f3f854e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48"
Oct 30 22:23:28.324833 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:28.306509    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjbf2\" (UniqueName: \"kubernetes.io/projected/cab0b42f-5ee8-4e8b-a6c7-75de81188465-kube-api-access-cjbf2\") pod \"payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29\" (UID: \"cab0b42f-5ee8-4e8b-a6c7-75de81188465\") " pod="openshift-ingress/payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29"
Oct 30 22:23:28.324833 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:28.324568    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48"
Oct 30 22:23:28.452370 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:28.452161    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:28.575665 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:28.575632    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29"
Oct 30 22:23:28.754258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:28.754136    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn"]
Oct 30 22:23:28.840887 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:28.840863    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48"]
Oct 30 22:23:28.841779 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:23:28.841751    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4378f9b0_0538_4d64_8697_9bfc2ac57027.slice/crio-b3e0f9a9a46538167d4d8704f35308dad3d5baeca5f9fe17deef941f0ddd2503 WatchSource:0}: Error finding container b3e0f9a9a46538167d4d8704f35308dad3d5baeca5f9fe17deef941f0ddd2503: Status 404 returned error can't find the container with id b3e0f9a9a46538167d4d8704f35308dad3d5baeca5f9fe17deef941f0ddd2503
Oct 30 22:23:29.078263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:29.078210    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:29.113729 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:23:29.113698    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab0b42f_5ee8_4e8b_a6c7_75de81188465.slice/crio-d44b9b2012562094184ae82f2d39f3eb8b58bbbe4a53e41d5452a84f40835ce0 WatchSource:0}: Error finding container d44b9b2012562094184ae82f2d39f3eb8b58bbbe4a53e41d5452a84f40835ce0: Status 404 returned error can't find the container with id d44b9b2012562094184ae82f2d39f3eb8b58bbbe4a53e41d5452a84f40835ce0
Oct 30 22:23:29.115582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:29.115559    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29"]
Oct 30 22:23:29.683249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:29.673531    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:29.843825 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:29.843795    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn" event={"ID":"4378f9b0-0538-4d64-8697-9bfc2ac57027","Type":"ContainerStarted","Data":"6df7006e9913e7984104b06f9602b424b39ee409ae08bf422b457656a2230830"}
Oct 30 22:23:29.843995 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:29.843862    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn"
Oct 30 22:23:29.843995 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:29.843886    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn" event={"ID":"4378f9b0-0538-4d64-8697-9bfc2ac57027","Type":"ContainerStarted","Data":"b3e0f9a9a46538167d4d8704f35308dad3d5baeca5f9fe17deef941f0ddd2503"}
Oct 30 22:23:29.845663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:29.845627    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48" event={"ID":"512d1cfc-681e-4360-aff3-791e3f3f854e","Type":"ContainerStarted","Data":"de506fd345026e7703e7a75a962b0af0ebcadf03924594542add45f08d7fe9ec"}
Oct 30 22:23:29.845846 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:29.845827    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48" event={"ID":"512d1cfc-681e-4360-aff3-791e3f3f854e","Type":"ContainerStarted","Data":"c38da5c4143912c95636b67591a5ccc80bdc3ff17544e987baddcd7003fe108f"}
Oct 30 22:23:29.845942 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:29.845863    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48"
Oct 30 22:23:29.847251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:29.847198    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29" event={"ID":"cab0b42f-5ee8-4e8b-a6c7-75de81188465","Type":"ContainerStarted","Data":"8782b8146c6b857c73f57c3f91086fa71d40a0ecb125cf10d019b29df6534a50"}
Oct 30 22:23:29.847359 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:29.847263    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29" event={"ID":"cab0b42f-5ee8-4e8b-a6c7-75de81188465","Type":"ContainerStarted","Data":"d44b9b2012562094184ae82f2d39f3eb8b58bbbe4a53e41d5452a84f40835ce0"}
Oct 30 22:23:29.847359 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:29.847287    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29"
Oct 30 22:23:29.885945 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:29.885897    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn" podStartSLOduration=2.8858851 podStartE2EDuration="2.8858851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:29.88508977 +0000 UTC m=+995.241456083" watchObservedRunningTime="2026-10-30 22:23:29.8858851 +0000 UTC m=+995.242251413"
Oct 30 22:23:29.927645 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:29.927600    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48" podStartSLOduration=2.927588263 podStartE2EDuration="2.927588263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:29.924758778 +0000 UTC m=+995.281125093" watchObservedRunningTime="2026-10-30 22:23:29.927588263 +0000 UTC m=+995.283954577"
Oct 30 22:23:29.968902 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:29.968799    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29" podStartSLOduration=2.968785977 podStartE2EDuration="2.968785977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:29.964640542 +0000 UTC m=+995.321006836" watchObservedRunningTime="2026-10-30 22:23:29.968785977 +0000 UTC m=+995.325152289"
Oct 30 22:23:30.527107 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:30.527082    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:30.714910 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:30.714873    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:31.601279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:31.596859    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:31.782560 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:31.782526    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:31.958995 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:31.958912    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:34.149038 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:34.148944    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 22:23:34.150080 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:34.150049    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.219562 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:34.219517    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.304251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:34.302329    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.339356 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:34.334438    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.546251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:34.545531    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.572265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:34.571363    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.656250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:34.640314    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.676310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:34.661327    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.676310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:34.662346    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.745373 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:34.741352    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.828309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:34.821328    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.168345 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:35.165635    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.168345 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:35.166065    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:23:35.303283 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:35.221636    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.303283 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:35.222062    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:23:35.324296 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:35.304579    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.324296 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:35.305008    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:23:35.371247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:35.338329    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.371247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:35.338734    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:23:35.420026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:35.419708    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.593995 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:35.559391    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.593995 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:35.571432    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:23:35.593995 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:35.576565    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.593995 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:35.580352    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:23:35.671334 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:35.667626    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.671334 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:35.668272    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.671334 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:35.669155    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:23:35.671334 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:35.669213    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:23:35.671334 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:35.670834    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 22:23:35.781821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:35.722330    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:23:36.397770 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:36.397725    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:36.450848 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:36.450744    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:36.485028 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:36.452560    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 22:23:36.485028 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:36.463551    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:23:36.485028 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:36.481076    4889 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:36.485028 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:36.481152    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg" podUID="7221740b-e003-4af1-86a2-396e32fff7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:36.561270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:36.559859    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:36.703592 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:36.703407    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:36.734251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:36.731008    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded"
Oct 30 22:23:36.762257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:36.754557    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:36.882533 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:36.882484    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:23:37.001481 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:37.001426    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:23:37.264298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:37.264188    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:37.431195 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:37.431152    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:37.452392 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:37.452338    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:37.482358 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:37.459142    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:37.482901 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:37.472899    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:37.618302 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:37.610666    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 22:23:37.758558 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:37.758512    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:37.784653 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:37.764350    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded"
Oct 30 22:23:37.904153 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:37.899735    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:38.073249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:38.068949    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:38.447249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:38.446334    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:38.484399 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:38.484349    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:38.484976 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:38.484946    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 22:23:38.500620 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:38.490190    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 22:23:38.625366 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:38.625311    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:38.692952 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:38.681332    4889 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:38.701516 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:38.701420    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt" podUID="3084be58-a1e9-410a-b501-2c71cca33742" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:38.709260 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:38.697038    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"]
Oct 30 22:23:38.823294 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:38.788346    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:38.823713 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:38.788384    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 22:23:38.906890 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:38.905557    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:23:38.916457 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:38.916392    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.040253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.036791    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe40461b-fff1-4d46-84d5-262b608028b2-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.066415 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.066378    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.066636 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.066615    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq4rn\" (UniqueName: \"kubernetes.io/projected/fe40461b-fff1-4d46-84d5-262b608028b2-kube-api-access-sq4rn\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.066815 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.066797    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.066960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.066943    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe40461b-fff1-4d46-84d5-262b608028b2-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.067147 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.067128    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.067297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.067278    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.071247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.067530    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.075257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.071452    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe40461b-fff1-4d46-84d5-262b608028b2-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.095249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.064426    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"]
Oct 30 22:23:39.095585 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.088726    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:23:39.096005 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.095377    4889 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:39.096183 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.096156    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2" podUID="3935b91c-508a-4e8d-99f2-23281921afba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:39.100246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.096806    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:23:39.186677 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.186620    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.186888 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.186681    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq4rn\" (UniqueName: \"kubernetes.io/projected/fe40461b-fff1-4d46-84d5-262b608028b2-kube-api-access-sq4rn\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.186888 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.186738    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.186888 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.186784    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe40461b-fff1-4d46-84d5-262b608028b2-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.186888 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.186878    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.187112 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.186909    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.187112 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.187041    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.187112 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.187094    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe40461b-fff1-4d46-84d5-262b608028b2-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.187297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.187275    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe40461b-fff1-4d46-84d5-262b608028b2-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.224248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.188710    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe40461b-fff1-4d46-84d5-262b608028b2-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.233254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.190855    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.320865 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.214565    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe40461b-fff1-4d46-84d5-262b608028b2-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.321135 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.214883    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.321358 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.223885    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.321504 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.234255    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.382594 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.382560    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq4rn\" (UniqueName: \"kubernetes.io/projected/fe40461b-fff1-4d46-84d5-262b608028b2-kube-api-access-sq4rn\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.383009 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.382987    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.399252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.391108    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe40461b-fff1-4d46-84d5-262b608028b2-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.455252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.442346    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:39.455252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.449739    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:39.518253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.496528    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:23:39.557256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.527452    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:39.752253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.752148    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 22:23:39.834461 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.826394    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 22:23:39.834461 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.828357    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded"
Oct 30 22:23:39.926608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:39.926563    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:40.115703 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:40.115582    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:23:40.468620 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:40.468511    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:23:40.505767 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:40.498478    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded"
Oct 30 22:23:40.550700 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:40.550643    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:23:40.568121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:40.568079    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:40.597053 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:40.596353    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 22:23:40.765399 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:40.765350    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:40.834332 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:40.834287    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded"
Oct 30 22:23:40.847760 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:40.847721    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:40.988388 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:40.955115    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:41.161304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:41.154328    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:41.161304 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:23:41.155142    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:23:41.421172 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:41.419298    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded"
Oct 30 22:23:41.477189 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:41.477144    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:23:41.511417 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:41.511358    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:41.535879 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:41.535838    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29"
Oct 30 22:23:41.536167 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:41.536150    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48"
Oct 30 22:23:41.536390 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:41.536377    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn"
Oct 30 22:23:41.536587 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:41.536555    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:23:41.536729 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:41.536718    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:23:41.550760 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:41.550709    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:23:41.551684 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:41.551651    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:23:41.603395 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:41.603347    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 22:23:41.648309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:41.647850    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:23:41.772152 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:41.771796    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 22:23:41.774483 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:41.774453    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:23:41.777106 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:41.777087    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:23:41.778903 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:41.777692    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:23:41.781104 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:41.781072    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:23:41.781258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:41.781195    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:23:41.781334 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:41.781314    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:23:41.790968 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:41.790947    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"]
Oct 30 22:23:41.791504 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:41.791472    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:23:42.531039 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:42.530994    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt" event={"ID":"fe40461b-fff1-4d46-84d5-262b608028b2","Type":"ContainerStarted","Data":"faef3ba24dd7105bccc2015733b748e403eacf8270ff455ccfc4a9521402e9a5"}
Oct 30 22:23:42.531647 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:42.531067    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt" event={"ID":"fe40461b-fff1-4d46-84d5-262b608028b2","Type":"ContainerStarted","Data":"e3b5bc6136144b4dcf18668f8400113d9a81d0d7aeb9bcd28455d02965fcf17d"}
Oct 30 22:23:42.582759 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:42.582701    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt" podStartSLOduration=4.582687144 podStartE2EDuration="4.582687144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:42.580351749 +0000 UTC m=+1007.936718063" watchObservedRunningTime="2026-10-30 22:23:42.582687144 +0000 UTC m=+1007.939053457"
Oct 30 22:23:42.891451 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:42.891336    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:43.098300 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:43.095741    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:43.449997 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:43.449967    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:43.451210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:43.451184    4889 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 22:23:43.451354 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:43.451264    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt" podUID="fe40461b-fff1-4d46-84d5-262b608028b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 22:23:44.450265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:44.450235    4889 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 22:23:44.450671 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:44.450288    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt" podUID="fe40461b-fff1-4d46-84d5-262b608028b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 22:23:44.911608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:44.911574    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:45.104059 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:45.104022    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:45.274131 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:45.274093    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:45.449990 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:45.449957    4889 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 22:23:45.450159 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:45.450011    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt" podUID="fe40461b-fff1-4d46-84d5-262b608028b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 22:23:46.362319 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:46.361770    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:46.450541 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:46.450512    4889 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 22:23:46.450682 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:46.450561    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt" podUID="fe40461b-fff1-4d46-84d5-262b608028b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 22:23:46.906988 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:46.906941    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:47.188455 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:47.187751    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:47.483306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:47.479376    4889 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 22:23:47.483306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:47.479450    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt" podUID="fe40461b-fff1-4d46-84d5-262b608028b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 22:23:48.212245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:48.207631    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:48.459758 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:48.459718    4889 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 22:23:48.459958 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:48.459776    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt" podUID="fe40461b-fff1-4d46-84d5-262b608028b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 22:23:49.109252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:49.105720    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:49.900023 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:49.899986    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:49.900610 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:49.900584    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:49.901612 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:49.901588    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:23:49.962530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:49.962495    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:50.816140 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:50.816109    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt"]
Oct 30 22:23:50.821466 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:50.821448    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt"
Oct 30 22:23:50.833054 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:50.833026    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/482c3f36-ae25-4f57-961f-20388f844f06-maas-api-tls\") pod \"maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt\" (UID: \"482c3f36-ae25-4f57-961f-20388f844f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt"
Oct 30 22:23:50.833182 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:50.833083    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/482c3f36-ae25-4f57-961f-20388f844f06-openshift-service-ca\") pod \"maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt\" (UID: \"482c3f36-ae25-4f57-961f-20388f844f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt"
Oct 30 22:23:50.833182 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:50.833121    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/482c3f36-ae25-4f57-961f-20388f844f06-metrics-certs\") pod \"maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt\" (UID: \"482c3f36-ae25-4f57-961f-20388f844f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt"
Oct 30 22:23:50.833182 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:50.833161    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqz47\" (UniqueName: \"kubernetes.io/projected/482c3f36-ae25-4f57-961f-20388f844f06-kube-api-access-sqz47\") pod \"maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt\" (UID: \"482c3f36-ae25-4f57-961f-20388f844f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt"
Oct 30 22:23:50.843970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:50.843933    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt"]
Oct 30 22:23:50.848163 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:50.848128    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-36d7ee39-serving-cert\""
Oct 30 22:23:50.934932 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:50.934900    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/482c3f36-ae25-4f57-961f-20388f844f06-maas-api-tls\") pod \"maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt\" (UID: \"482c3f36-ae25-4f57-961f-20388f844f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt"
Oct 30 22:23:50.935118 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:50.934953    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/482c3f36-ae25-4f57-961f-20388f844f06-openshift-service-ca\") pod \"maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt\" (UID: \"482c3f36-ae25-4f57-961f-20388f844f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt"
Oct 30 22:23:50.935118 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:50.934981    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/482c3f36-ae25-4f57-961f-20388f844f06-metrics-certs\") pod \"maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt\" (UID: \"482c3f36-ae25-4f57-961f-20388f844f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt"
Oct 30 22:23:50.935118 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:50.935012    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqz47\" (UniqueName: \"kubernetes.io/projected/482c3f36-ae25-4f57-961f-20388f844f06-kube-api-access-sqz47\") pod \"maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt\" (UID: \"482c3f36-ae25-4f57-961f-20388f844f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt"
Oct 30 22:23:50.935118 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:23:50.935035    4889 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-serving-cert: secret "maas-api-e2e-derive-36d7ee39-serving-cert" not found
Oct 30 22:23:50.935118 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:23:50.935107    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/482c3f36-ae25-4f57-961f-20388f844f06-maas-api-tls podName:482c3f36-ae25-4f57-961f-20388f844f06 nodeName:}" failed. No retries permitted until 2026-10-30 22:23:51.435090481 +0000 UTC m=+1016.791456773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/482c3f36-ae25-4f57-961f-20388f844f06-maas-api-tls") pod "maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt" (UID: "482c3f36-ae25-4f57-961f-20388f844f06") : secret "maas-api-e2e-derive-36d7ee39-serving-cert" not found
Oct 30 22:23:50.935596 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:50.935578    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/482c3f36-ae25-4f57-961f-20388f844f06-openshift-service-ca\") pod \"maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt\" (UID: \"482c3f36-ae25-4f57-961f-20388f844f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt"
Oct 30 22:23:50.937569 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:50.937545    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/482c3f36-ae25-4f57-961f-20388f844f06-metrics-certs\") pod \"maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt\" (UID: \"482c3f36-ae25-4f57-961f-20388f844f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt"
Oct 30 22:23:50.955670 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:50.955637    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqz47\" (UniqueName: \"kubernetes.io/projected/482c3f36-ae25-4f57-961f-20388f844f06-kube-api-access-sqz47\") pod \"maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt\" (UID: \"482c3f36-ae25-4f57-961f-20388f844f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt"
Oct 30 22:23:50.969423 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:50.969392    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:51.151335 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:51.151086    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:51.315848 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:51.315817    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:51.457937 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:51.457850    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/482c3f36-ae25-4f57-961f-20388f844f06-maas-api-tls\") pod \"maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt\" (UID: \"482c3f36-ae25-4f57-961f-20388f844f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt"
Oct 30 22:23:51.460516 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:51.460482    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/482c3f36-ae25-4f57-961f-20388f844f06-maas-api-tls\") pod \"maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt\" (UID: \"482c3f36-ae25-4f57-961f-20388f844f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt"
Oct 30 22:23:51.742634 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:51.742603    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt"
Oct 30 22:23:52.248938 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:52.248906    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt"]
Oct 30 22:23:52.255642 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:52.255029    4889 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:23:52.806678 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:52.806641    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt" event={"ID":"482c3f36-ae25-4f57-961f-20388f844f06","Type":"ContainerStarted","Data":"c56cdfe89a06c01072fcc40ea125a9a908f396ab98b9daab3dad52e242217b7f"}
Oct 30 22:23:52.806846 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:52.806702    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt" event={"ID":"482c3f36-ae25-4f57-961f-20388f844f06","Type":"ContainerStarted","Data":"9cf62da4583498d486ed4b6e53e16e17bebdda230f6dd1267f603064d2a68210"}
Oct 30 22:23:52.806846 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:52.806734    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt"
Oct 30 22:23:52.848407 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:52.848362    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt" podStartSLOduration=2.443359087 podStartE2EDuration="2.84835127s" totalImagesPullingTime="404.992183ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:52.847718003 +0000 UTC m=+1018.204084316" watchObservedRunningTime="2026-10-30 22:23:52.84835127 +0000 UTC m=+1018.204717583"
Oct 30 22:23:53.067257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:53.063857    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:53.745658 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:53.745624    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:53.919385 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:53.919349    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:54.086723 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:54.086637    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:54.247703 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:54.247668    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:54.812865 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:54.812833    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:55.412990 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:55.412960    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:56.582397 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:56.574424    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:58.341270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:58.338643    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:58.820202 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:58.820167    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt"
Oct 30 22:23:59.933251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:23:59.919905    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:00.633147 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:00.633110    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:00.806945 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:00.806910    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:00.969818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:00.969698    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:01.193495 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:01.193451    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:01.690284 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:01.689171    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:02.329451 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:02.329413    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:03.436370 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:03.436289    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:05.149342 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:05.144672    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:07.331941 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:07.331902    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:08.040254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:08.038305    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:08.214376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:08.214343    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:09.449424 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:09.449378    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:24:09.491256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:09.489106    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:09.520573 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:09.520529    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:09.805250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:09.804810    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 22:24:10.428249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:10.427319    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:10.472258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:10.470709    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:10.472258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:10.471345    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:24:10.502262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:10.501346    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:10.502262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:10.502030    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:24:10.502526 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:10.502303    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:10.540256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:10.526268    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:10.540256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:10.526634    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:24:10.756395 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:10.756331    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 22:24:10.854666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:10.854616    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:10.948703 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:24:10.858965    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:24:10.993260 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:10.903605    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:11.054511 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:11.054383    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:11.087627 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:11.087578    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:11.313304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:11.285655    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:24:11.535392 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:11.529389    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:11.601528 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:11.577088    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:11.601528 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:11.588542    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:11.601528 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:11.590163    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:11.681024 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:11.664282    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded"
Oct 30 22:24:11.850250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:11.848376    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:11.868485 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:11.868376    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:12.125306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:12.120328    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:12.214318 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:12.210591    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:12.484378 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:12.326189    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:24:12.484378 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:12.358027    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:12.629348 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:12.629285    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:24:12.652914 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:12.650864    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:24:12.872334 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:24:12.870099    4889 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.28s"
Oct 30 22:24:12.872334 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:12.870179    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:24:12.992463 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:12.992336    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 22:24:13.086872 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:13.086362    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:24:13.086872 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:13.086440    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:24:13.086872 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:13.086554    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:24:13.086872 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:13.086592    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:24:13.220298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:13.219312    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:13.225783 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:13.224410    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 22:24:13.249561 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:13.233330    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:13.249561 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:13.233567    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 22:24:13.249561 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:13.234145    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded"
Oct 30 22:24:13.269425 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:13.269343    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 22:24:13.378929 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:13.378357    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:24:13.422445 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:13.421297    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:24:13.628248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:13.625441    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:13.652114 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:13.651354    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:24:13.664254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:13.653325    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:13.743238 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:13.743140    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:13.842336 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:13.842295    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:13.914863 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:13.914766    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:14.158252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:14.154802    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 22:24:14.231296 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:14.230282    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:14.255261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:14.243023    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:14.255261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:14.243472    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:14.255261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:14.243824    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:14.380579 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:14.380529    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:14.652824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:14.652461    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:14.675416 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:14.675374    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:14.838082 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:14.838036    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:14.874123 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:14.843468    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": context deadline exceeded"
Oct 30 22:24:15.120250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:15.120117    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:24:15.156250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:15.155280    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:24:15.329269 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:15.296553    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:15.329269 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:15.296930    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:24:15.329269 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:15.297203    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded"
Oct 30 22:24:15.329269 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:15.297548    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 22:24:15.380330 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:15.372315    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:15.416264 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:15.402262    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:15.416264 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:15.411045    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 22:24:15.672062 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:15.667724    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 22:24:15.684240 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:15.682055    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:24:15.690347 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:15.689408    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 22:24:15.690500 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:15.690443    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6"
Oct 30 22:24:15.806267 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:15.805418    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 22:24:16.560437 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:16.560401    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:18.746028 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:18.745992    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:20.948246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:20.945868    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:23.058791 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:23.049976    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:23.221498 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:23.221387    4889 generic.go:356] "Generic (PLEG): container finished" podID="50b1a3f3-f888-408b-b6ac-548b3eef0ca9" containerID="0f4b43efc19244f8b918488e9dc25979f2fe451315913cb960ce8761639f25a6" exitCode=0
Oct 30 22:24:23.221498 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:23.221444    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m" event={"ID":"50b1a3f3-f888-408b-b6ac-548b3eef0ca9","Type":"ContainerDied","Data":"0f4b43efc19244f8b918488e9dc25979f2fe451315913cb960ce8761639f25a6"}
Oct 30 22:24:23.221881 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:23.221866    4889 scope.go:122] "RemoveContainer" containerID="0f4b43efc19244f8b918488e9dc25979f2fe451315913cb960ce8761639f25a6"
Oct 30 22:24:23.224286 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:23.224259    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:24:23.224394 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:23.224330    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:24:23.224394 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:23.224362    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:24:23.567406 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:23.567325    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:24:24.243035 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:24.242996    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m" event={"ID":"50b1a3f3-f888-408b-b6ac-548b3eef0ca9","Type":"ContainerStarted","Data":"e80ce05c1caa51fa63fc0f1e8a95bbb14e175db596bc741ce44da46ed16642aa"}
Oct 30 22:24:24.322477 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:24.314693    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:24:24.322477 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:24.314840    4889 patch_prober.go:29] interesting pod/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 22:24:24.322477 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:24.314899    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m" podUID="50b1a3f3-f888-408b-b6ac-548b3eef0ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 22:24:25.313263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:25.307503    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:25.321267 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:25.315751    4889 patch_prober.go:29] interesting pod/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 22:24:25.321267 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:25.315822    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m" podUID="50b1a3f3-f888-408b-b6ac-548b3eef0ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 22:24:25.826250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:25.817784    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:26.328683 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:26.328632    4889 patch_prober.go:29] interesting pod/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 22:24:26.329303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:26.329270    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m" podUID="50b1a3f3-f888-408b-b6ac-548b3eef0ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 22:24:26.437854 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:26.437820    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29"]
Oct 30 22:24:26.459747 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:26.459715    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48"]
Oct 30 22:24:26.635281 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:26.635137    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:26.886518 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:26.878595    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:27.316247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:27.314458    4889 patch_prober.go:29] interesting pod/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 22:24:27.316247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:27.314530    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m" podUID="50b1a3f3-f888-408b-b6ac-548b3eef0ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 22:24:27.799687 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:27.799654    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:28.333365 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:28.333331    4889 patch_prober.go:29] interesting pod/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 22:24:28.334305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:28.334271    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m" podUID="50b1a3f3-f888-408b-b6ac-548b3eef0ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 22:24:29.329250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:29.318628    4889 patch_prober.go:29] interesting pod/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 22:24:29.329250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:29.318689    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m" podUID="50b1a3f3-f888-408b-b6ac-548b3eef0ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 22:24:29.738216 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:29.738109    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:30.320510 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:30.320481    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:24:30.320960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:30.320930    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:24:30.321957 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:30.321936    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:24:30.408302 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:30.408273    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:30.570459 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:30.570426    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:31.181895 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:31.181861    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn"]
Oct 30 22:24:31.392270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:31.392237    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:31.481297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:31.458335    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29" podUID="cab0b42f-5ee8-4e8b-a6c7-75de81188465" containerName="payload-processing" containerID="cri-o://8782b8146c6b857c73f57c3f91086fa71d40a0ecb125cf10d019b29df6534a50" gracePeriod=25
Oct 30 22:24:31.531431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:31.467299    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48" podUID="512d1cfc-681e-4360-aff3-791e3f3f854e" containerName="payload-pre-processing" containerID="cri-o://de506fd345026e7703e7a75a962b0af0ebcadf03924594542add45f08d7fe9ec" gracePeriod=25
Oct 30 22:24:31.576471 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:31.564966    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:31.837884 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:31.837852    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:31.875550 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:31.875526    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29"
Oct 30 22:24:31.899019 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:31.898999    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48"
Oct 30 22:24:32.023592 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:31.984323    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/512d1cfc-681e-4360-aff3-791e3f3f854e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/512d1cfc-681e-4360-aff3-791e3f3f854e-plugins-config-volume\") pod \"512d1cfc-681e-4360-aff3-791e3f3f854e\" (UID: \"512d1cfc-681e-4360-aff3-791e3f3f854e\") "
Oct 30 22:24:32.023592 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:31.984600    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cab0b42f-5ee8-4e8b-a6c7-75de81188465-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab0b42f-5ee8-4e8b-a6c7-75de81188465-plugins-config-volume\") pod \"cab0b42f-5ee8-4e8b-a6c7-75de81188465\" (UID: \"cab0b42f-5ee8-4e8b-a6c7-75de81188465\") "
Oct 30 22:24:32.023592 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:31.984838    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cab0b42f-5ee8-4e8b-a6c7-75de81188465-kube-api-access-cjbf2\" (UniqueName: \"kubernetes.io/projected/cab0b42f-5ee8-4e8b-a6c7-75de81188465-kube-api-access-cjbf2\") pod \"cab0b42f-5ee8-4e8b-a6c7-75de81188465\" (UID: \"cab0b42f-5ee8-4e8b-a6c7-75de81188465\") "
Oct 30 22:24:32.023592 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:31.984905    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/512d1cfc-681e-4360-aff3-791e3f3f854e-kube-api-access-v77nk\" (UniqueName: \"kubernetes.io/projected/512d1cfc-681e-4360-aff3-791e3f3f854e-kube-api-access-v77nk\") pod \"512d1cfc-681e-4360-aff3-791e3f3f854e\" (UID: \"512d1cfc-681e-4360-aff3-791e3f3f854e\") "
Oct 30 22:24:32.023592 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:31.999444    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/512d1cfc-681e-4360-aff3-791e3f3f854e-plugins-config-volume" pod "512d1cfc-681e-4360-aff3-791e3f3f854e" (UID: "512d1cfc-681e-4360-aff3-791e3f3f854e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:32.023592 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:32.002364    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cab0b42f-5ee8-4e8b-a6c7-75de81188465-plugins-config-volume" pod "cab0b42f-5ee8-4e8b-a6c7-75de81188465" (UID: "cab0b42f-5ee8-4e8b-a6c7-75de81188465"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:32.109746 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:32.093689    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/512d1cfc-681e-4360-aff3-791e3f3f854e-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:24:32.109746 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:32.093726    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab0b42f-5ee8-4e8b-a6c7-75de81188465-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:24:32.116607 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:32.116540    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab0b42f-5ee8-4e8b-a6c7-75de81188465-kube-api-access-cjbf2" pod "cab0b42f-5ee8-4e8b-a6c7-75de81188465" (UID: "cab0b42f-5ee8-4e8b-a6c7-75de81188465"). InnerVolumeSpecName "kube-api-access-cjbf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:32.119583 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:32.119529    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/512d1cfc-681e-4360-aff3-791e3f3f854e-kube-api-access-v77nk" pod "512d1cfc-681e-4360-aff3-791e3f3f854e" (UID: "512d1cfc-681e-4360-aff3-791e3f3f854e"). InnerVolumeSpecName "kube-api-access-v77nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:32.233853 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:32.196016    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjbf2\" (UniqueName: \"kubernetes.io/projected/cab0b42f-5ee8-4e8b-a6c7-75de81188465-kube-api-access-cjbf2\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:24:32.233853 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:32.196048    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v77nk\" (UniqueName: \"kubernetes.io/projected/512d1cfc-681e-4360-aff3-791e3f3f854e-kube-api-access-v77nk\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:24:32.396504 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:32.396424    4889 generic.go:356] "Generic (PLEG): container finished" podID="cab0b42f-5ee8-4e8b-a6c7-75de81188465" containerID="8782b8146c6b857c73f57c3f91086fa71d40a0ecb125cf10d019b29df6534a50" exitCode=0
Oct 30 22:24:32.396504 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:32.396472    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29"
Oct 30 22:24:32.396722 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:32.396512    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29" event={"ID":"cab0b42f-5ee8-4e8b-a6c7-75de81188465","Type":"ContainerDied","Data":"8782b8146c6b857c73f57c3f91086fa71d40a0ecb125cf10d019b29df6534a50"}
Oct 30 22:24:32.396722 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:32.396564    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29" event={"ID":"cab0b42f-5ee8-4e8b-a6c7-75de81188465","Type":"ContainerDied","Data":"d44b9b2012562094184ae82f2d39f3eb8b58bbbe4a53e41d5452a84f40835ce0"}
Oct 30 22:24:32.396722 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:32.396587    4889 scope.go:122] "RemoveContainer" containerID="8782b8146c6b857c73f57c3f91086fa71d40a0ecb125cf10d019b29df6534a50"
Oct 30 22:24:32.398235 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:32.398179    4889 generic.go:356] "Generic (PLEG): container finished" podID="512d1cfc-681e-4360-aff3-791e3f3f854e" containerID="de506fd345026e7703e7a75a962b0af0ebcadf03924594542add45f08d7fe9ec" exitCode=0
Oct 30 22:24:32.398235 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:32.398219    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48"
Oct 30 22:24:32.398416 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:32.398251    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48" event={"ID":"512d1cfc-681e-4360-aff3-791e3f3f854e","Type":"ContainerDied","Data":"de506fd345026e7703e7a75a962b0af0ebcadf03924594542add45f08d7fe9ec"}
Oct 30 22:24:32.398416 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:32.398300    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48" event={"ID":"512d1cfc-681e-4360-aff3-791e3f3f854e","Type":"ContainerDied","Data":"c38da5c4143912c95636b67591a5ccc80bdc3ff17544e987baddcd7003fe108f"}
Oct 30 22:24:32.408142 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:32.408123    4889 scope.go:122] "RemoveContainer" containerID="8782b8146c6b857c73f57c3f91086fa71d40a0ecb125cf10d019b29df6534a50"
Oct 30 22:24:32.408404 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:24:32.408378    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8782b8146c6b857c73f57c3f91086fa71d40a0ecb125cf10d019b29df6534a50\": container with ID starting with 8782b8146c6b857c73f57c3f91086fa71d40a0ecb125cf10d019b29df6534a50 not found: ID does not exist" containerID="8782b8146c6b857c73f57c3f91086fa71d40a0ecb125cf10d019b29df6534a50"
Oct 30 22:24:32.408484 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:32.408403    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8782b8146c6b857c73f57c3f91086fa71d40a0ecb125cf10d019b29df6534a50"} err="failed to get container status \"8782b8146c6b857c73f57c3f91086fa71d40a0ecb125cf10d019b29df6534a50\": rpc error: code = NotFound desc = could not find container \"8782b8146c6b857c73f57c3f91086fa71d40a0ecb125cf10d019b29df6534a50\": container with ID starting with 8782b8146c6b857c73f57c3f91086fa71d40a0ecb125cf10d019b29df6534a50 not found: ID does not exist"
Oct 30 22:24:32.408484 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:32.408417    4889 scope.go:122] "RemoveContainer" containerID="de506fd345026e7703e7a75a962b0af0ebcadf03924594542add45f08d7fe9ec"
Oct 30 22:24:32.417838 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:32.417817    4889 scope.go:122] "RemoveContainer" containerID="de506fd345026e7703e7a75a962b0af0ebcadf03924594542add45f08d7fe9ec"
Oct 30 22:24:32.418127 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:24:32.418107    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de506fd345026e7703e7a75a962b0af0ebcadf03924594542add45f08d7fe9ec\": container with ID starting with de506fd345026e7703e7a75a962b0af0ebcadf03924594542add45f08d7fe9ec not found: ID does not exist" containerID="de506fd345026e7703e7a75a962b0af0ebcadf03924594542add45f08d7fe9ec"
Oct 30 22:24:32.418254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:32.418131    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de506fd345026e7703e7a75a962b0af0ebcadf03924594542add45f08d7fe9ec"} err="failed to get container status \"de506fd345026e7703e7a75a962b0af0ebcadf03924594542add45f08d7fe9ec\": rpc error: code = NotFound desc = could not find container \"de506fd345026e7703e7a75a962b0af0ebcadf03924594542add45f08d7fe9ec\": container with ID starting with de506fd345026e7703e7a75a962b0af0ebcadf03924594542add45f08d7fe9ec not found: ID does not exist"
Oct 30 22:24:32.450356 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:32.450326    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48"]
Oct 30 22:24:32.454320 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:32.454301    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ba9ae08-6847f996c5-n5b48"]
Oct 30 22:24:32.496065 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:32.496031    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29"]
Oct 30 22:24:32.503080 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:32.503057    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ba9ae08-5fdd496c5c-4sz29"]
Oct 30 22:24:32.776417 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:32.776388    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="512d1cfc-681e-4360-aff3-791e3f3f854e" path="/var/lib/kubelet/pods/512d1cfc-681e-4360-aff3-791e3f3f854e/volumes"
Oct 30 22:24:32.776897 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:32.776883    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cab0b42f-5ee8-4e8b-a6c7-75de81188465" path="/var/lib/kubelet/pods/cab0b42f-5ee8-4e8b-a6c7-75de81188465/volumes"
Oct 30 22:24:33.409443 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:33.409409    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:34.081249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:34.077008    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:34.256250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:34.255511    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:34.462301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:34.440646    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:34.641962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:34.640481    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:35.113870 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:35.113819    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:35.293807 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:35.293772    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:35.469343 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:35.467361    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:35.638250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:35.635762    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:35.901466 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:35.901436    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"]
Oct 30 22:24:35.902113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:35.902095    4889 cpu_manager.go:414] "containerMap: removing container" podUID="512d1cfc-681e-4360-aff3-791e3f3f854e" containerName="payload-pre-processing"
Oct 30 22:24:35.902113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:35.902112    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="512d1cfc-681e-4360-aff3-791e3f3f854e" containerName="payload-pre-processing"
Oct 30 22:24:35.902346 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:35.902161    4889 cpu_manager.go:414] "containerMap: removing container" podUID="cab0b42f-5ee8-4e8b-a6c7-75de81188465" containerName="payload-processing"
Oct 30 22:24:35.902346 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:35.902167    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab0b42f-5ee8-4e8b-a6c7-75de81188465" containerName="payload-processing"
Oct 30 22:24:35.902346 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:35.902287    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="512d1cfc-681e-4360-aff3-791e3f3f854e" containerName="payload-pre-processing"
Oct 30 22:24:35.902346 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:35.902307    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="cab0b42f-5ee8-4e8b-a6c7-75de81188465" containerName="payload-processing"
Oct 30 22:24:35.906028 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:35.906007    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:35.932917 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:35.932892    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"]
Oct 30 22:24:35.938341 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:35.938317    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-476e593d-gw-openshift-default-dockercfg-hlqvg\""
Oct 30 22:24:35.961988 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:35.961955    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-envoy\") pod \"e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") " pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:35.962159 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:35.962124    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istiod-ca-cert\") pod \"e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") " pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:35.962267 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:35.962179    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dnkz\" (UniqueName: \"kubernetes.io/projected/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-kube-api-access-7dnkz\") pod \"e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") " pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:35.962343 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:35.962306    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-credential-socket\") pod \"e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") " pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:35.962434 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:35.962406    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-data\") pod \"e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") " pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:35.962488 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:35.962473    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-workload-certs\") pod \"e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") " pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:35.962544 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:35.962509    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-podinfo\") pod \"e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") " pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:35.962713 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:35.962687    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-workload-socket\") pod \"e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") " pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:35.962785 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:35.962728    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-token\") pod \"e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") " pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:36.020113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.020075    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:36.064645 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.064616    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istiod-ca-cert\") pod \"e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") " pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:36.064824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.064692    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dnkz\" (UniqueName: \"kubernetes.io/projected/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-kube-api-access-7dnkz\") pod \"e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") " pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:36.064824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.064753    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-credential-socket\") pod \"e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") " pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:36.064824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.064792    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-data\") pod \"e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") " pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:36.065167 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.064826    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-workload-certs\") pod \"e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") " pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:36.065167 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.064848    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-podinfo\") pod \"e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") " pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:36.065300 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.065184    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-credential-socket\") pod \"e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") " pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:36.065357 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.065291    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-data\") pod \"e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") " pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:36.065357 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.065315    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-workload-certs\") pod \"e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") " pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:36.065463 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.065406    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istiod-ca-cert\") pod \"e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") " pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:36.065505 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.065486    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-workload-socket\") pod \"e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") " pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:36.065547 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.065523    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-token\") pod \"e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") " pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:36.065584 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.065577    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-envoy\") pod \"e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") " pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:36.065745 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.065728    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-workload-socket\") pod \"e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") " pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:36.067721 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.067698    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-podinfo\") pod \"e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") " pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:36.067799 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.067737    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-envoy\") pod \"e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") " pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:36.141250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.109637    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-token\") pod \"e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") " pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:36.186454 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.186358    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn" podUID="4378f9b0-0538-4d64-8697-9bfc2ac57027" containerName="payload-processing" containerID="cri-o://6df7006e9913e7984104b06f9602b424b39ee409ae08bf422b457656a2230830" gracePeriod=25
Oct 30 22:24:36.274707 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.235312    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:36.274707 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.265901    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dnkz\" (UniqueName: \"kubernetes.io/projected/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-kube-api-access-7dnkz\") pod \"e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") " pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:36.534413 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.534372    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:36.585280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.585189    4889 generic.go:356] "Generic (PLEG): container finished" podID="4378f9b0-0538-4d64-8697-9bfc2ac57027" containerID="6df7006e9913e7984104b06f9602b424b39ee409ae08bf422b457656a2230830" exitCode=0
Oct 30 22:24:36.585517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.585491    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn" event={"ID":"4378f9b0-0538-4d64-8697-9bfc2ac57027","Type":"ContainerDied","Data":"6df7006e9913e7984104b06f9602b424b39ee409ae08bf422b457656a2230830"}
Oct 30 22:24:36.789074 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.789041    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn"
Oct 30 22:24:36.918251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.912192    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:37.035390 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.965808    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4378f9b0-0538-4d64-8697-9bfc2ac57027-kube-api-access-fb4m6\" (UniqueName: \"kubernetes.io/projected/4378f9b0-0538-4d64-8697-9bfc2ac57027-kube-api-access-fb4m6\") pod \"4378f9b0-0538-4d64-8697-9bfc2ac57027\" (UID: \"4378f9b0-0538-4d64-8697-9bfc2ac57027\") "
Oct 30 22:24:37.035390 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:36.966162    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4378f9b0-0538-4d64-8697-9bfc2ac57027-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4378f9b0-0538-4d64-8697-9bfc2ac57027-plugins-config-volume\") pod \"4378f9b0-0538-4d64-8697-9bfc2ac57027\" (UID: \"4378f9b0-0538-4d64-8697-9bfc2ac57027\") "
Oct 30 22:24:37.039261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:37.035686    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4378f9b0-0538-4d64-8697-9bfc2ac57027-plugins-config-volume" pod "4378f9b0-0538-4d64-8697-9bfc2ac57027" (UID: "4378f9b0-0538-4d64-8697-9bfc2ac57027"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:37.093643 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:37.089742    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4378f9b0-0538-4d64-8697-9bfc2ac57027-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:24:37.101276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:37.100799    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4378f9b0-0538-4d64-8697-9bfc2ac57027-kube-api-access-fb4m6" pod "4378f9b0-0538-4d64-8697-9bfc2ac57027" (UID: "4378f9b0-0538-4d64-8697-9bfc2ac57027"). InnerVolumeSpecName "kube-api-access-fb4m6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:37.126250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:37.120027    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"]
Oct 30 22:24:37.197197 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:37.197140    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb4m6\" (UniqueName: \"kubernetes.io/projected/4378f9b0-0538-4d64-8697-9bfc2ac57027-kube-api-access-fb4m6\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:24:37.227251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:37.222291    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:24:37.227251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:37.222387    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:24:37.227251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:37.222431    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:24:37.287089 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:37.280291    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt"]
Oct 30 22:24:37.287089 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:37.280527    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt" podUID="482c3f36-ae25-4f57-961f-20388f844f06" containerName="maas-api" containerID="cri-o://c56cdfe89a06c01072fcc40ea125a9a908f396ab98b9daab3dad52e242217b7f" gracePeriod=30
Oct 30 22:24:37.361056 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:37.361027    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:37.537504 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:37.537375    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:37.725088 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:37.725051    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn" event={"ID":"4378f9b0-0538-4d64-8697-9bfc2ac57027","Type":"ContainerDied","Data":"b3e0f9a9a46538167d4d8704f35308dad3d5baeca5f9fe17deef941f0ddd2503"}
Oct 30 22:24:37.725355 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:37.725336    4889 scope.go:122] "RemoveContainer" containerID="6df7006e9913e7984104b06f9602b424b39ee409ae08bf422b457656a2230830"
Oct 30 22:24:37.725650 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:37.725633    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn"
Oct 30 22:24:37.795191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:37.795105    4889 generic.go:356] "Generic (PLEG): container finished" podID="482c3f36-ae25-4f57-961f-20388f844f06" containerID="c56cdfe89a06c01072fcc40ea125a9a908f396ab98b9daab3dad52e242217b7f" exitCode=0
Oct 30 22:24:37.798566 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:37.798535    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v" event={"ID":"e967d2cf-50fa-4e9e-90b9-d49da1b81da1","Type":"ContainerStarted","Data":"4910b367e11072bf49e3bd3713ce4a895c12dd1cbda0c5d28756284360e66482"}
Oct 30 22:24:37.801468 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:37.801425    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v" event={"ID":"e967d2cf-50fa-4e9e-90b9-d49da1b81da1","Type":"ContainerStarted","Data":"63f25645a8b1336cdae5fe4a14c8b60282761abfcd39457f19d534c19dbcab2e"}
Oct 30 22:24:37.801665 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:37.801640    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt" event={"ID":"482c3f36-ae25-4f57-961f-20388f844f06","Type":"ContainerDied","Data":"c56cdfe89a06c01072fcc40ea125a9a908f396ab98b9daab3dad52e242217b7f"}
Oct 30 22:24:37.864250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:37.857263    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v" podStartSLOduration=2.857241007 podStartE2EDuration="2.857241007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:37.854953076 +0000 UTC m=+1063.211319392" watchObservedRunningTime="2026-10-30 22:24:37.857241007 +0000 UTC m=+1063.213607321"
Oct 30 22:24:37.939515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:37.939384    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn"]
Oct 30 22:24:37.957755 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:37.957498    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt"
Oct 30 22:24:37.960536 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:37.960507    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-36d7ee39-5cf5d67547-bbzkn"]
Oct 30 22:24:38.062331 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.061261    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/482c3f36-ae25-4f57-961f-20388f844f06-metrics-certs\" (UniqueName: \"kubernetes.io/secret/482c3f36-ae25-4f57-961f-20388f844f06-metrics-certs\") pod \"482c3f36-ae25-4f57-961f-20388f844f06\" (UID: \"482c3f36-ae25-4f57-961f-20388f844f06\") "
Oct 30 22:24:38.121262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.117388    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/482c3f36-ae25-4f57-961f-20388f844f06-kube-api-access-sqz47\" (UniqueName: \"kubernetes.io/projected/482c3f36-ae25-4f57-961f-20388f844f06-kube-api-access-sqz47\") pod \"482c3f36-ae25-4f57-961f-20388f844f06\" (UID: \"482c3f36-ae25-4f57-961f-20388f844f06\") "
Oct 30 22:24:38.121262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.117465    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/482c3f36-ae25-4f57-961f-20388f844f06-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/482c3f36-ae25-4f57-961f-20388f844f06-maas-api-tls\") pod \"482c3f36-ae25-4f57-961f-20388f844f06\" (UID: \"482c3f36-ae25-4f57-961f-20388f844f06\") "
Oct 30 22:24:38.121262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.117554    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/482c3f36-ae25-4f57-961f-20388f844f06-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/482c3f36-ae25-4f57-961f-20388f844f06-openshift-service-ca\") pod \"482c3f36-ae25-4f57-961f-20388f844f06\" (UID: \"482c3f36-ae25-4f57-961f-20388f844f06\") "
Oct 30 22:24:38.121262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.118891    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/482c3f36-ae25-4f57-961f-20388f844f06-openshift-service-ca" pod "482c3f36-ae25-4f57-961f-20388f844f06" (UID: "482c3f36-ae25-4f57-961f-20388f844f06"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:38.143374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.122285    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:38.143374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.122434    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"]
Oct 30 22:24:38.143374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.123487    4889 cpu_manager.go:414] "containerMap: removing container" podUID="4378f9b0-0538-4d64-8697-9bfc2ac57027" containerName="payload-processing"
Oct 30 22:24:38.143374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.123505    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4378f9b0-0538-4d64-8697-9bfc2ac57027" containerName="payload-processing"
Oct 30 22:24:38.143374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.123576    4889 cpu_manager.go:414] "containerMap: removing container" podUID="482c3f36-ae25-4f57-961f-20388f844f06" containerName="maas-api"
Oct 30 22:24:38.143374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.123584    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="482c3f36-ae25-4f57-961f-20388f844f06" containerName="maas-api"
Oct 30 22:24:38.143374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.123729    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="482c3f36-ae25-4f57-961f-20388f844f06" containerName="maas-api"
Oct 30 22:24:38.143374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.123750    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="4378f9b0-0538-4d64-8697-9bfc2ac57027" containerName="payload-processing"
Oct 30 22:24:38.143374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.131092    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/482c3f36-ae25-4f57-961f-20388f844f06-maas-api-tls" pod "482c3f36-ae25-4f57-961f-20388f844f06" (UID: "482c3f36-ae25-4f57-961f-20388f844f06"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:38.143374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.143190    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/482c3f36-ae25-4f57-961f-20388f844f06-kube-api-access-sqz47" pod "482c3f36-ae25-4f57-961f-20388f844f06" (UID: "482c3f36-ae25-4f57-961f-20388f844f06"). InnerVolumeSpecName "kube-api-access-sqz47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:38.157342 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.144506    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.180503 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.160638    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-913eddcb-openshift-default-dockercfg-22lj5\""
Oct 30 22:24:38.206373 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.205667    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"]
Oct 30 22:24:38.325418 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.324991    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:38.326946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.326883    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqz47\" (UniqueName: \"kubernetes.io/projected/482c3f36-ae25-4f57-961f-20388f844f06-kube-api-access-sqz47\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:24:38.326946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.326910    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/482c3f36-ae25-4f57-961f-20388f844f06-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:24:38.326946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.326925    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/482c3f36-ae25-4f57-961f-20388f844f06-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:24:38.335493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.335437    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/482c3f36-ae25-4f57-961f-20388f844f06-metrics-certs" pod "482c3f36-ae25-4f57-961f-20388f844f06" (UID: "482c3f36-ae25-4f57-961f-20388f844f06"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:38.471266 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.462068    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-data\") pod \"e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") " pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.471266 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.462243    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-workload-socket\") pod \"e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") " pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.474261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.472450    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-credential-socket\") pod \"e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") " pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.474261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.472672    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istiod-ca-cert\") pod \"e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") " pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.474261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.472780    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-workload-certs\") pod \"e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") " pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.474261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.472810    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-podinfo\") pod \"e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") " pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.474261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.472864    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-envoy\") pod \"e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") " pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.474261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.473096    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-token\") pod \"e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") " pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.474261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.473122    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsc9s\" (UniqueName: \"kubernetes.io/projected/f1c26fbd-ffd1-438b-8c6a-36b667be335c-kube-api-access-fsc9s\") pod \"e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") " pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.474261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.473335    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/482c3f36-ae25-4f57-961f-20388f844f06-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:24:38.537888 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.537844    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:38.547762 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.547734    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:38.548084 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.548065    4889 patch_prober.go:29] interesting pod/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 22:24:38.548593 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.548561    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v" podUID="e967d2cf-50fa-4e9e-90b9-d49da1b81da1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 22:24:38.589210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.582164    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-data\") pod \"e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") " pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.589210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.582310    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-workload-socket\") pod \"e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") " pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.589210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.582388    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-credential-socket\") pod \"e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") " pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.589210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.582465    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istiod-ca-cert\") pod \"e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") " pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.589210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.582523    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-workload-certs\") pod \"e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") " pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.589210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.582544    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-podinfo\") pod \"e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") " pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.589210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.582589    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-envoy\") pod \"e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") " pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.589210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.582744    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-token\") pod \"e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") " pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.589210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.582769    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsc9s\" (UniqueName: \"kubernetes.io/projected/f1c26fbd-ffd1-438b-8c6a-36b667be335c-kube-api-access-fsc9s\") pod \"e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") " pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.589210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.583583    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-data\") pod \"e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") " pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.589210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.583847    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-workload-socket\") pod \"e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") " pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.589210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.584076    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-credential-socket\") pod \"e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") " pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.607253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.599275    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-workload-certs\") pod \"e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") " pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.627663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.626436    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istiod-ca-cert\") pod \"e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") " pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.730840 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.701791    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-podinfo\") pod \"e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") " pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.784698 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.734197    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"]
Oct 30 22:24:38.785697 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.774163    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:38.789986 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.786027    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m" podUID="50b1a3f3-f888-408b-b6ac-548b3eef0ca9" containerName="istio-proxy" containerID="cri-o://e80ce05c1caa51fa63fc0f1e8a95bbb14e175db596bc741ce44da46ed16642aa" gracePeriod=30
Oct 30 22:24:38.817352 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.813027    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4378f9b0-0538-4d64-8697-9bfc2ac57027" path="/var/lib/kubelet/pods/4378f9b0-0538-4d64-8697-9bfc2ac57027/volumes"
Oct 30 22:24:38.828387 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.820534    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-envoy\") pod \"e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") " pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.828387 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.821194    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsc9s\" (UniqueName: \"kubernetes.io/projected/f1c26fbd-ffd1-438b-8c6a-36b667be335c-kube-api-access-fsc9s\") pod \"e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") " pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.828387 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.821599    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-token\") pod \"e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") " pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:38.855288 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.850447    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt"
Oct 30 22:24:38.856536 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.855908    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt" event={"ID":"482c3f36-ae25-4f57-961f-20388f844f06","Type":"ContainerDied","Data":"9cf62da4583498d486ed4b6e53e16e17bebdda230f6dd1267f603064d2a68210"}
Oct 30 22:24:38.856536 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:38.855978    4889 scope.go:122] "RemoveContainer" containerID="c56cdfe89a06c01072fcc40ea125a9a908f396ab98b9daab3dad52e242217b7f"
Oct 30 22:24:39.025203 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:39.025170    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt"]
Oct 30 22:24:39.026292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:39.025949    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:39.079496 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:39.057917    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-36d7ee39-5545d6fcb6-bfwmt"]
Oct 30 22:24:39.540022 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:39.539991    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:39.562972 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:39.562941    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:39.620394 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:24:39.620349    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c26fbd_ffd1_438b_8c6a_36b667be335c.slice/crio-032779cab05dee3f739d3c841dd2b0fb13c93c73bee7c817d23a68639434a672 WatchSource:0}: Error finding container 032779cab05dee3f739d3c841dd2b0fb13c93c73bee7c817d23a68639434a672: Status 404 returned error can't find the container with id 032779cab05dee3f739d3c841dd2b0fb13c93c73bee7c817d23a68639434a672
Oct 30 22:24:39.621397 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:39.621171    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"]
Oct 30 22:24:39.624117 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:39.624089    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:24:39.624205 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:39.624154    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:24:39.624205 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:39.624184    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:24:39.747783 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:39.747748    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:39.862855 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:39.862758    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz" event={"ID":"f1c26fbd-ffd1-438b-8c6a-36b667be335c","Type":"ContainerStarted","Data":"cd0aac5fcff56bd903844172a57158ecb1d1e322ce3189f3cc2da4dc1ceebeac"}
Oct 30 22:24:39.862855 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:39.862826    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz" event={"ID":"f1c26fbd-ffd1-438b-8c6a-36b667be335c","Type":"ContainerStarted","Data":"032779cab05dee3f739d3c841dd2b0fb13c93c73bee7c817d23a68639434a672"}
Oct 30 22:24:39.863844 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:39.863527    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:39.864742 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:39.864720    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:24:39.917710 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:39.917659    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz" podStartSLOduration=1.917640252 podStartE2EDuration="1.917640252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:39.914122703 +0000 UTC m=+1065.270489042" watchObservedRunningTime="2026-10-30 22:24:39.917640252 +0000 UTC m=+1065.274006568"
Oct 30 22:24:40.036958 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:40.036925    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:40.038253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:40.038162    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:40.043291 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:40.043270    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:40.252958 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:40.252912    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:40.303959 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:24:40.303927    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:24:40.778906 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:40.778868    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="482c3f36-ae25-4f57-961f-20388f844f06" path="/var/lib/kubelet/pods/482c3f36-ae25-4f57-961f-20388f844f06/volumes"
Oct 30 22:24:40.872143 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:40.872111    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:40.873086 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:40.873065    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:24:40.897342 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:40.897315    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:40.946987 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:40.946954    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f"]
Oct 30 22:24:40.951522 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:40.951497    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f"
Oct 30 22:24:40.973703 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:40.973672    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-476e593d\""
Oct 30 22:24:40.985805 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:40.979177    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f"]
Oct 30 22:24:41.019555 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:41.019512    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8"]
Oct 30 22:24:41.023921 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:41.023896    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8"
Oct 30 22:24:41.043774 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:41.043749    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-476e593d-dockercfg-l7lh4\""
Oct 30 22:24:41.044571 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:41.044551    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8"]
Oct 30 22:24:41.087698 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:41.087660    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4j5h\" (UniqueName: \"kubernetes.io/projected/2aa8b895-0530-439f-b390-042fe1c36f28-kube-api-access-r4j5h\") pod \"payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f\" (UID: \"2aa8b895-0530-439f-b390-042fe1c36f28\") " pod="openshift-ingress/payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f"
Oct 30 22:24:41.087869 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:41.087741    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvz9f\" (UniqueName: \"kubernetes.io/projected/b3a4495e-2010-4901-8c4b-f495f8b2374e-kube-api-access-nvz9f\") pod \"payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8\" (UID: \"b3a4495e-2010-4901-8c4b-f495f8b2374e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8"
Oct 30 22:24:41.087869 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:41.087850    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aa8b895-0530-439f-b390-042fe1c36f28-plugins-config-volume\") pod \"payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f\" (UID: \"2aa8b895-0530-439f-b390-042fe1c36f28\") " pod="openshift-ingress/payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f"
Oct 30 22:24:41.088057 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:41.088030    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3a4495e-2010-4901-8c4b-f495f8b2374e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8\" (UID: \"b3a4495e-2010-4901-8c4b-f495f8b2374e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8"
Oct 30 22:24:41.090547 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:41.090520    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:41.201247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:41.197446    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4j5h\" (UniqueName: \"kubernetes.io/projected/2aa8b895-0530-439f-b390-042fe1c36f28-kube-api-access-r4j5h\") pod \"payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f\" (UID: \"2aa8b895-0530-439f-b390-042fe1c36f28\") " pod="openshift-ingress/payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f"
Oct 30 22:24:41.201247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:41.197563    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvz9f\" (UniqueName: \"kubernetes.io/projected/b3a4495e-2010-4901-8c4b-f495f8b2374e-kube-api-access-nvz9f\") pod \"payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8\" (UID: \"b3a4495e-2010-4901-8c4b-f495f8b2374e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8"
Oct 30 22:24:41.201247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:41.197643    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aa8b895-0530-439f-b390-042fe1c36f28-plugins-config-volume\") pod \"payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f\" (UID: \"2aa8b895-0530-439f-b390-042fe1c36f28\") " pod="openshift-ingress/payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f"
Oct 30 22:24:41.201247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:41.197806    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3a4495e-2010-4901-8c4b-f495f8b2374e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8\" (UID: \"b3a4495e-2010-4901-8c4b-f495f8b2374e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8"
Oct 30 22:24:41.237093 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:41.226564    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aa8b895-0530-439f-b390-042fe1c36f28-plugins-config-volume\") pod \"payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f\" (UID: \"2aa8b895-0530-439f-b390-042fe1c36f28\") " pod="openshift-ingress/payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f"
Oct 30 22:24:41.248250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:41.244553    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3a4495e-2010-4901-8c4b-f495f8b2374e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8\" (UID: \"b3a4495e-2010-4901-8c4b-f495f8b2374e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8"
Oct 30 22:24:41.292298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:41.291767    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvz9f\" (UniqueName: \"kubernetes.io/projected/b3a4495e-2010-4901-8c4b-f495f8b2374e-kube-api-access-nvz9f\") pod \"payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8\" (UID: \"b3a4495e-2010-4901-8c4b-f495f8b2374e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8"
Oct 30 22:24:41.312196 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:41.312122    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4j5h\" (UniqueName: \"kubernetes.io/projected/2aa8b895-0530-439f-b390-042fe1c36f28-kube-api-access-r4j5h\") pod \"payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f\" (UID: \"2aa8b895-0530-439f-b390-042fe1c36f28\") " pod="openshift-ingress/payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f"
Oct 30 22:24:41.320481 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:41.315955    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:41.341808 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:41.341541    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8"
Oct 30 22:24:41.594261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:41.586450    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f"
Oct 30 22:24:41.879858 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:24:41.879826    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a4495e_2010_4901_8c4b_f495f8b2374e.slice/crio-24ac3e13a584cfaf3b94be05897f29b5a8a96e4f448ce552769a69ce84b2513d WatchSource:0}: Error finding container 24ac3e13a584cfaf3b94be05897f29b5a8a96e4f448ce552769a69ce84b2513d: Status 404 returned error can't find the container with id 24ac3e13a584cfaf3b94be05897f29b5a8a96e4f448ce552769a69ce84b2513d
Oct 30 22:24:41.880833 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:41.880745    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8"]
Oct 30 22:24:41.929306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:41.929278    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f"]
Oct 30 22:24:41.938630 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:24:41.938605    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aa8b895_0530_439f_b390_042fe1c36f28.slice/crio-263ada31897eb2010eb9aba2318f1e6881bb39e8987614a66e8dd86f0cf4749a WatchSource:0}: Error finding container 263ada31897eb2010eb9aba2318f1e6881bb39e8987614a66e8dd86f0cf4749a: Status 404 returned error can't find the container with id 263ada31897eb2010eb9aba2318f1e6881bb39e8987614a66e8dd86f0cf4749a
Oct 30 22:24:41.950732 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:41.950635    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:42.117209 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:42.117177    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:42.889852 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:42.889814    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f" event={"ID":"2aa8b895-0530-439f-b390-042fe1c36f28","Type":"ContainerStarted","Data":"3c79fa09a8c8830c8672623d0e9b559254cc1f1e12b939d36f98720f13db464b"}
Oct 30 22:24:42.890380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:42.889876    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f" event={"ID":"2aa8b895-0530-439f-b390-042fe1c36f28","Type":"ContainerStarted","Data":"263ada31897eb2010eb9aba2318f1e6881bb39e8987614a66e8dd86f0cf4749a"}
Oct 30 22:24:42.890380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:42.889904    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f"
Oct 30 22:24:42.891521 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:42.891490    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8" event={"ID":"b3a4495e-2010-4901-8c4b-f495f8b2374e","Type":"ContainerStarted","Data":"158274f9bbc1eb68357b750e021b7dcd65ca48b16b89ed519c3ed4191a04dc83"}
Oct 30 22:24:42.891640 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:42.891537    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8" event={"ID":"b3a4495e-2010-4901-8c4b-f495f8b2374e","Type":"ContainerStarted","Data":"24ac3e13a584cfaf3b94be05897f29b5a8a96e4f448ce552769a69ce84b2513d"}
Oct 30 22:24:42.891640 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:42.891557    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8"
Oct 30 22:24:42.934385 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:42.934335    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f" podStartSLOduration=2.934322624 podStartE2EDuration="2.934322624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:42.930177812 +0000 UTC m=+1068.286544122" watchObservedRunningTime="2026-10-30 22:24:42.934322624 +0000 UTC m=+1068.290688938"
Oct 30 22:24:42.972739 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:42.972689    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8" podStartSLOduration=2.97266492 podStartE2EDuration="2.97266492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:42.969311151 +0000 UTC m=+1068.325677466" watchObservedRunningTime="2026-10-30 22:24:42.97266492 +0000 UTC m=+1068.329031212"
Oct 30 22:24:43.208320 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:43.206042    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:43.614304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:43.614271    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:43.736235 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:43.736191    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242"]
Oct 30 22:24:43.739864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:43.739845    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242"
Oct 30 22:24:43.760703 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:43.760675    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242"]
Oct 30 22:24:43.768134 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:43.768112    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-913eddcb-dockercfg-t85zb\""
Oct 30 22:24:43.768287 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:43.768113    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-913eddcb\""
Oct 30 22:24:43.857248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:43.850686    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:43.910027 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:43.879680    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wspg5\" (UniqueName: \"kubernetes.io/projected/e7b8353e-019c-4ebf-9a34-771262f281e9-kube-api-access-wspg5\") pod \"payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242\" (UID: \"e7b8353e-019c-4ebf-9a34-771262f281e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242"
Oct 30 22:24:43.910027 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:43.879972    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7b8353e-019c-4ebf-9a34-771262f281e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242\" (UID: \"e7b8353e-019c-4ebf-9a34-771262f281e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242"
Oct 30 22:24:44.023265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.012778    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wspg5\" (UniqueName: \"kubernetes.io/projected/e7b8353e-019c-4ebf-9a34-771262f281e9-kube-api-access-wspg5\") pod \"payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242\" (UID: \"e7b8353e-019c-4ebf-9a34-771262f281e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242"
Oct 30 22:24:44.023265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.013074    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7b8353e-019c-4ebf-9a34-771262f281e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242\" (UID: \"e7b8353e-019c-4ebf-9a34-771262f281e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242"
Oct 30 22:24:44.084247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.079205    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7b8353e-019c-4ebf-9a34-771262f281e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242\" (UID: \"e7b8353e-019c-4ebf-9a34-771262f281e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242"
Oct 30 22:24:44.202757 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.202709    4889 generic.go:356] "Generic (PLEG): container finished" podID="50b1a3f3-f888-408b-b6ac-548b3eef0ca9" containerID="e80ce05c1caa51fa63fc0f1e8a95bbb14e175db596bc741ce44da46ed16642aa" exitCode=0
Oct 30 22:24:44.204357 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.204325    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m" event={"ID":"50b1a3f3-f888-408b-b6ac-548b3eef0ca9","Type":"ContainerDied","Data":"e80ce05c1caa51fa63fc0f1e8a95bbb14e175db596bc741ce44da46ed16642aa"}
Oct 30 22:24:44.215626 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.215599    4889 scope.go:122] "RemoveContainer" containerID="0f4b43efc19244f8b918488e9dc25979f2fe451315913cb960ce8761639f25a6"
Oct 30 22:24:44.230297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.226323    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wspg5\" (UniqueName: \"kubernetes.io/projected/e7b8353e-019c-4ebf-9a34-771262f281e9-kube-api-access-wspg5\") pod \"payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242\" (UID: \"e7b8353e-019c-4ebf-9a34-771262f281e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242"
Oct 30 22:24:44.240532 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.238915    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z"]
Oct 30 22:24:44.248251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.244499    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z"
Oct 30 22:24:44.248251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.247746    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z"]
Oct 30 22:24:44.286054 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.285575    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn5jt\" (UniqueName: \"kubernetes.io/projected/0f1b6c42-a7aa-468b-9974-09e2e80338c5-kube-api-access-jn5jt\") pod \"payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z\" (UID: \"0f1b6c42-a7aa-468b-9974-09e2e80338c5\") " pod="openshift-ingress/payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z"
Oct 30 22:24:44.286054 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.286001    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1b6c42-a7aa-468b-9974-09e2e80338c5-plugins-config-volume\") pod \"payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z\" (UID: \"0f1b6c42-a7aa-468b-9974-09e2e80338c5\") " pod="openshift-ingress/payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z"
Oct 30 22:24:44.358472 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.358435    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242"
Oct 30 22:24:44.393738 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.393701    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1b6c42-a7aa-468b-9974-09e2e80338c5-plugins-config-volume\") pod \"payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z\" (UID: \"0f1b6c42-a7aa-468b-9974-09e2e80338c5\") " pod="openshift-ingress/payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z"
Oct 30 22:24:44.393881 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.393771    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn5jt\" (UniqueName: \"kubernetes.io/projected/0f1b6c42-a7aa-468b-9974-09e2e80338c5-kube-api-access-jn5jt\") pod \"payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z\" (UID: \"0f1b6c42-a7aa-468b-9974-09e2e80338c5\") " pod="openshift-ingress/payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z"
Oct 30 22:24:44.394775 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.394712    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1b6c42-a7aa-468b-9974-09e2e80338c5-plugins-config-volume\") pod \"payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z\" (UID: \"0f1b6c42-a7aa-468b-9974-09e2e80338c5\") " pod="openshift-ingress/payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z"
Oct 30 22:24:44.415541 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.415512    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn5jt\" (UniqueName: \"kubernetes.io/projected/0f1b6c42-a7aa-468b-9974-09e2e80338c5-kube-api-access-jn5jt\") pod \"payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z\" (UID: \"0f1b6c42-a7aa-468b-9974-09e2e80338c5\") " pod="openshift-ingress/payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z"
Oct 30 22:24:44.442090 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.442056    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:24:44.494675 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.494638    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-credential-socket\") pod \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") "
Oct 30 22:24:44.494868 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.494757    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-token\" (UniqueName: \"kubernetes.io/projected/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-token\") pod \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") "
Oct 30 22:24:44.495084 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.495051    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-credential-socket" pod "50b1a3f3-f888-408b-b6ac-548b3eef0ca9" (UID: "50b1a3f3-f888-408b-b6ac-548b3eef0ca9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:44.495524 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.495503    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-envoy\") pod \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") "
Oct 30 22:24:44.495621 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.495563    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-kube-api-access-8vzrl\" (UniqueName: \"kubernetes.io/projected/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-kube-api-access-8vzrl\") pod \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") "
Oct 30 22:24:44.495702 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.495686    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-data\") pod \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") "
Oct 30 22:24:44.495760 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.495752    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istiod-ca-cert\") pod \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") "
Oct 30 22:24:44.495820 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.495784    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-workload-socket\") pod \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") "
Oct 30 22:24:44.495896 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.495881    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-podinfo\") pod \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") "
Oct 30 22:24:44.495961 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.495925    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-workload-certs\") pod \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\" (UID: \"50b1a3f3-f888-408b-b6ac-548b3eef0ca9\") "
Oct 30 22:24:44.496441 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.496417    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-data" pod "50b1a3f3-f888-408b-b6ac-548b3eef0ca9" (UID: "50b1a3f3-f888-408b-b6ac-548b3eef0ca9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:44.497004 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.496976    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istiod-ca-cert" pod "50b1a3f3-f888-408b-b6ac-548b3eef0ca9" (UID: "50b1a3f3-f888-408b-b6ac-548b3eef0ca9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:44.497198 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.497172    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-workload-certs" pod "50b1a3f3-f888-408b-b6ac-548b3eef0ca9" (UID: "50b1a3f3-f888-408b-b6ac-548b3eef0ca9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:44.497553 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.497526    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:24:44.497643 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.497558    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:24:44.497643 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.497571    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:24:44.497643 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.497581    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:24:44.497832 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.497651    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-workload-socket" pod "50b1a3f3-f888-408b-b6ac-548b3eef0ca9" (UID: "50b1a3f3-f888-408b-b6ac-548b3eef0ca9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:44.499755 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.499719    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-token" pod "50b1a3f3-f888-408b-b6ac-548b3eef0ca9" (UID: "50b1a3f3-f888-408b-b6ac-548b3eef0ca9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:44.500538 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.500510    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-kube-api-access-8vzrl" pod "50b1a3f3-f888-408b-b6ac-548b3eef0ca9" (UID: "50b1a3f3-f888-408b-b6ac-548b3eef0ca9"). InnerVolumeSpecName "kube-api-access-8vzrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:44.501319 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.501296    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-podinfo" pod "50b1a3f3-f888-408b-b6ac-548b3eef0ca9" (UID: "50b1a3f3-f888-408b-b6ac-548b3eef0ca9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:24:44.501387 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.501345    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-envoy" pod "50b1a3f3-f888-408b-b6ac-548b3eef0ca9" (UID: "50b1a3f3-f888-408b-b6ac-548b3eef0ca9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:44.575912 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.575868    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z"
Oct 30 22:24:44.599207 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.599176    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:24:44.599207 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.599204    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:24:44.599207 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.599213    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:24:44.599410 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.599239    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vzrl\" (UniqueName: \"kubernetes.io/projected/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-kube-api-access-8vzrl\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:24:44.599410 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.599250    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b1a3f3-f888-408b-b6ac-548b3eef0ca9-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:24:44.707514 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.707468    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242"]
Oct 30 22:24:44.883039 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:24:44.883003    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f1b6c42_a7aa_468b_9974_09e2e80338c5.slice/crio-d1939907f74584c4feac72ece45e566b6025b74483707dca53a40b900d53e0d3 WatchSource:0}: Error finding container d1939907f74584c4feac72ece45e566b6025b74483707dca53a40b900d53e0d3: Status 404 returned error can't find the container with id d1939907f74584c4feac72ece45e566b6025b74483707dca53a40b900d53e0d3
Oct 30 22:24:44.884214 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:44.884194    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z"]
Oct 30 22:24:45.127247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:45.123660    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:45.252845 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:45.252814    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"
Oct 30 22:24:45.253040 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:45.252956    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m" event={"ID":"50b1a3f3-f888-408b-b6ac-548b3eef0ca9","Type":"ContainerDied","Data":"3a78d8e4540bb54373e08039be067fe23f6b86cca63c4ef2cb4835aec75eebbc"}
Oct 30 22:24:45.253040 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:45.253014    4889 scope.go:122] "RemoveContainer" containerID="e80ce05c1caa51fa63fc0f1e8a95bbb14e175db596bc741ce44da46ed16642aa"
Oct 30 22:24:45.258338 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:45.258284    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z" event={"ID":"0f1b6c42-a7aa-468b-9974-09e2e80338c5","Type":"ContainerStarted","Data":"ea05da12c1b9d68fdd78055c8ca711de91f20d03cf6076717f143f23112a9a65"}
Oct 30 22:24:45.259178 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:45.258366    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z" event={"ID":"0f1b6c42-a7aa-468b-9974-09e2e80338c5","Type":"ContainerStarted","Data":"d1939907f74584c4feac72ece45e566b6025b74483707dca53a40b900d53e0d3"}
Oct 30 22:24:45.259300 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:45.259218    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z"
Oct 30 22:24:45.260799 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:45.260771    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242" event={"ID":"e7b8353e-019c-4ebf-9a34-771262f281e9","Type":"ContainerStarted","Data":"55076776392fa44c52d98346f2192799b98066b22b75fea591bd01168a533a47"}
Oct 30 22:24:45.260883 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:45.260815    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242" event={"ID":"e7b8353e-019c-4ebf-9a34-771262f281e9","Type":"ContainerStarted","Data":"b7a9ac614e698d74234d87f385b477a84b623625c29a435712b9749987304b86"}
Oct 30 22:24:45.260991 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:45.260969    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242"
Oct 30 22:24:45.307449 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:45.307419    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"]
Oct 30 22:24:45.319557 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:45.319533    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-36d7ee39-openshift-default-6b7cbf646f-nnh2m"]
Oct 30 22:24:45.358798 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:45.358754    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242" podStartSLOduration=2.358742749 podStartE2EDuration="2.358742749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:45.356216436 +0000 UTC m=+1070.712582749" watchObservedRunningTime="2026-10-30 22:24:45.358742749 +0000 UTC m=+1070.715109062"
Oct 30 22:24:45.395542 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:45.395498    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z" podStartSLOduration=2.395486711 podStartE2EDuration="2.395486711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:45.3943025 +0000 UTC m=+1070.750668814" watchObservedRunningTime="2026-10-30 22:24:45.395486711 +0000 UTC m=+1070.751853024"
Oct 30 22:24:46.480054 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:46.480004    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:46.776021 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:46.775991    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50b1a3f3-f888-408b-b6ac-548b3eef0ca9" path="/var/lib/kubelet/pods/50b1a3f3-f888-408b-b6ac-548b3eef0ca9/volumes"
Oct 30 22:24:46.835756 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:46.835717    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:48.662250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:48.656337    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:50.446415 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:50.438110    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:52.144604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:52.144519    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:53.793895 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:53.793853    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:53.961760 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:53.961709    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:54.208487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:54.208403    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8"
Oct 30 22:24:54.221616 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:54.221590    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f"
Oct 30 22:24:55.627874 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:55.627835    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:56.272764 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:56.272734    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242"
Oct 30 22:24:57.199868 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:57.199836    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt"]
Oct 30 22:24:57.200619 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:57.200597    4889 cpu_manager.go:414] "containerMap: removing container" podUID="50b1a3f3-f888-408b-b6ac-548b3eef0ca9" containerName="istio-proxy"
Oct 30 22:24:57.200619 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:57.200616    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b1a3f3-f888-408b-b6ac-548b3eef0ca9" containerName="istio-proxy"
Oct 30 22:24:57.200791 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:57.200735    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b1a3f3-f888-408b-b6ac-548b3eef0ca9" containerName="istio-proxy"
Oct 30 22:24:57.204255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:57.204237    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt"
Oct 30 22:24:57.221509 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:57.221483    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt"]
Oct 30 22:24:57.229965 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:57.229939    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-913eddcb-serving-cert\""
Oct 30 22:24:57.279621 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:57.279592    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z"
Oct 30 22:24:57.352426 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:57.352390    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:57.366599 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:57.366568    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjqdr\" (UniqueName: \"kubernetes.io/projected/31ca0794-c411-43dc-bfab-589ba56c79cf-kube-api-access-wjqdr\") pod \"maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt\" (UID: \"31ca0794-c411-43dc-bfab-589ba56c79cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt"
Oct 30 22:24:57.366768 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:57.366643    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31ca0794-c411-43dc-bfab-589ba56c79cf-metrics-certs\") pod \"maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt\" (UID: \"31ca0794-c411-43dc-bfab-589ba56c79cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt"
Oct 30 22:24:57.366768 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:57.366749    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31ca0794-c411-43dc-bfab-589ba56c79cf-maas-api-tls\") pod \"maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt\" (UID: \"31ca0794-c411-43dc-bfab-589ba56c79cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt"
Oct 30 22:24:57.366961 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:57.366935    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ca0794-c411-43dc-bfab-589ba56c79cf-openshift-service-ca\") pod \"maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt\" (UID: \"31ca0794-c411-43dc-bfab-589ba56c79cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt"
Oct 30 22:24:57.469298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:57.468434    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjqdr\" (UniqueName: \"kubernetes.io/projected/31ca0794-c411-43dc-bfab-589ba56c79cf-kube-api-access-wjqdr\") pod \"maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt\" (UID: \"31ca0794-c411-43dc-bfab-589ba56c79cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt"
Oct 30 22:24:57.469298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:57.468528    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31ca0794-c411-43dc-bfab-589ba56c79cf-metrics-certs\") pod \"maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt\" (UID: \"31ca0794-c411-43dc-bfab-589ba56c79cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt"
Oct 30 22:24:57.469298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:57.468614    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31ca0794-c411-43dc-bfab-589ba56c79cf-maas-api-tls\") pod \"maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt\" (UID: \"31ca0794-c411-43dc-bfab-589ba56c79cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt"
Oct 30 22:24:57.469298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:57.468774    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ca0794-c411-43dc-bfab-589ba56c79cf-openshift-service-ca\") pod \"maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt\" (UID: \"31ca0794-c411-43dc-bfab-589ba56c79cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt"
Oct 30 22:24:57.489252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:57.486436    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ca0794-c411-43dc-bfab-589ba56c79cf-openshift-service-ca\") pod \"maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt\" (UID: \"31ca0794-c411-43dc-bfab-589ba56c79cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt"
Oct 30 22:24:57.512255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:57.506626    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31ca0794-c411-43dc-bfab-589ba56c79cf-metrics-certs\") pod \"maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt\" (UID: \"31ca0794-c411-43dc-bfab-589ba56c79cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt"
Oct 30 22:24:57.533246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:57.524619    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31ca0794-c411-43dc-bfab-589ba56c79cf-maas-api-tls\") pod \"maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt\" (UID: \"31ca0794-c411-43dc-bfab-589ba56c79cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt"
Oct 30 22:24:57.570286 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:57.570046    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjqdr\" (UniqueName: \"kubernetes.io/projected/31ca0794-c411-43dc-bfab-589ba56c79cf-kube-api-access-wjqdr\") pod \"maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt\" (UID: \"31ca0794-c411-43dc-bfab-589ba56c79cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt"
Oct 30 22:24:57.584249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:57.570853    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:57.818567 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:57.818533    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt"
Oct 30 22:24:58.169411 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:58.169381    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt"]
Oct 30 22:24:58.169884 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:24:58.169856    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ca0794_c411_43dc_bfab_589ba56c79cf.slice/crio-2083f1bd3ab76129befc0a10a25b6469e33afd454fa900a6d46006720f541c52 WatchSource:0}: Error finding container 2083f1bd3ab76129befc0a10a25b6469e33afd454fa900a6d46006720f541c52: Status 404 returned error can't find the container with id 2083f1bd3ab76129befc0a10a25b6469e33afd454fa900a6d46006720f541c52
Oct 30 22:24:58.465356 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:58.465260    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt" event={"ID":"31ca0794-c411-43dc-bfab-589ba56c79cf","Type":"ContainerStarted","Data":"2083f1bd3ab76129befc0a10a25b6469e33afd454fa900a6d46006720f541c52"}
Oct 30 22:24:59.221074 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:59.221025    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:59.475085 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:59.474994    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt" event={"ID":"31ca0794-c411-43dc-bfab-589ba56c79cf","Type":"ContainerStarted","Data":"8d248ccb2e8f98da7c060363f504baf32a7ac808827acbd850bc7358f210500b"}
Oct 30 22:24:59.475085 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:59.475049    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt"
Oct 30 22:24:59.517026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:24:59.516973    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt" podStartSLOduration=2.057248295 podStartE2EDuration="2.516956375s" totalImagesPullingTime="459.70808ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:59.514823064 +0000 UTC m=+1084.871189379" watchObservedRunningTime="2026-10-30 22:24:59.516956375 +0000 UTC m=+1084.873322730"
Oct 30 22:25:01.046253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:01.043109    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:02.756248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:02.748291    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:04.525522 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:04.525477    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:05.487949 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:05.487915    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt"
Oct 30 22:25:06.556929 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:06.548404    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:08.533066 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:08.531950    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:10.322066 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:10.322027    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:10.428475 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:25:10.428407    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:25:12.078273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:12.076746    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:13.026206 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:13.026165    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:13.241525 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:13.241474    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:13.740997 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:13.740965    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:15.591262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:15.586942    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:17.800307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:17.798868    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:19.766266 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:19.762482    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:21.536251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:21.528964    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:23.188251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:23.178140    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:23.864062 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:23.864024    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:26.028246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:26.018756    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:26.040708 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:26.040681    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f"]
Oct 30 22:25:26.081259 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:26.075638    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8"]
Oct 30 22:25:26.216258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:26.203713    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:26.482255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:26.481999    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:28.097249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:28.096198    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:30.157434 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:30.157399    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:30.499998 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:30.499970    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:30.902032 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:30.901547    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"]
Oct 30 22:25:30.904278 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:30.902678    4889 cpu_manager.go:414] "containerMap: removing container" podUID="50b1a3f3-f888-408b-b6ac-548b3eef0ca9" containerName="istio-proxy"
Oct 30 22:25:30.904278 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:30.902703    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b1a3f3-f888-408b-b6ac-548b3eef0ca9" containerName="istio-proxy"
Oct 30 22:25:30.904278 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:30.902940    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b1a3f3-f888-408b-b6ac-548b3eef0ca9" containerName="istio-proxy"
Oct 30 22:25:30.907361 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:30.907337    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:30.970361 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:30.968152    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2949c143-openshift-default-dockercfg-z8qs7\""
Oct 30 22:25:31.077282 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.028134    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istiod-ca-cert\") pod \"e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") " pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.077282 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.028379    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-envoy\") pod \"e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") " pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.077282 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.044475    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f" podUID="2aa8b895-0530-439f-b390-042fe1c36f28" containerName="payload-processing" containerID="cri-o://3c79fa09a8c8830c8672623d0e9b559254cc1f1e12b939d36f98720f13db464b" gracePeriod=25
Oct 30 22:25:31.079086 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.078767    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxsbm\" (UniqueName: \"kubernetes.io/projected/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-kube-api-access-cxsbm\") pod \"e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") " pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.079086 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.078956    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-credential-socket\") pod \"e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") " pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.079086 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.079008    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-workload-certs\") pod \"e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") " pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.079404 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.079369    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-data\") pod \"e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") " pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.079712 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.079642    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-workload-socket\") pod \"e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") " pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.079712 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.079681    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-token\") pod \"e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") " pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.080034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.079929    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-podinfo\") pod \"e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") " pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.085674 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.085616    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8" podUID="b3a4495e-2010-4901-8c4b-f495f8b2374e" containerName="payload-pre-processing" containerID="cri-o://158274f9bbc1eb68357b750e021b7dcd65ca48b16b89ed519c3ed4191a04dc83" gracePeriod=25
Oct 30 22:25:31.168936 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.156524    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"]
Oct 30 22:25:31.211741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.198697    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:31.211741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.200126    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-credential-socket\") pod \"e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") " pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.211741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.200183    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-workload-certs\") pod \"e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") " pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.211741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.200277    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-data\") pod \"e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") " pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.211741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.200330    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-workload-socket\") pod \"e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") " pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.211741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.200377    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-token\") pod \"e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") " pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.211741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.200427    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-podinfo\") pod \"e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") " pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.211741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.200502    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istiod-ca-cert\") pod \"e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") " pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.211741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.200727    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-envoy\") pod \"e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") " pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.211741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.200744    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-credential-socket\") pod \"e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") " pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.211741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.200755    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxsbm\" (UniqueName: \"kubernetes.io/projected/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-kube-api-access-cxsbm\") pod \"e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") " pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.211741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.201353    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-workload-certs\") pod \"e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") " pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.211741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.201585    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-data\") pod \"e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") " pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.211741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.201815    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-workload-socket\") pod \"e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") " pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.211741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.202536    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istiod-ca-cert\") pod \"e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") " pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.225513 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.224453    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxsbm\" (UniqueName: \"kubernetes.io/projected/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-kube-api-access-cxsbm\") pod \"e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") " pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.227137 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.226967    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-podinfo\") pod \"e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") " pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.241502 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.241425    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-token\") pod \"e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") " pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.352244 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.348957    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-envoy\") pod \"e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") " pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.405248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.402905    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:31.626305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.526079    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:31.824040 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.821052    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8"
Oct 30 22:25:31.848986 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.848450    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3a4495e-2010-4901-8c4b-f495f8b2374e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3a4495e-2010-4901-8c4b-f495f8b2374e-plugins-config-volume\") pod \"b3a4495e-2010-4901-8c4b-f495f8b2374e\" (UID: \"b3a4495e-2010-4901-8c4b-f495f8b2374e\") "
Oct 30 22:25:31.848986 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.848566    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3a4495e-2010-4901-8c4b-f495f8b2374e-kube-api-access-nvz9f\" (UniqueName: \"kubernetes.io/projected/b3a4495e-2010-4901-8c4b-f495f8b2374e-kube-api-access-nvz9f\") pod \"b3a4495e-2010-4901-8c4b-f495f8b2374e\" (UID: \"b3a4495e-2010-4901-8c4b-f495f8b2374e\") "
Oct 30 22:25:31.849400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.849355    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a4495e-2010-4901-8c4b-f495f8b2374e-plugins-config-volume" pod "b3a4495e-2010-4901-8c4b-f495f8b2374e" (UID: "b3a4495e-2010-4901-8c4b-f495f8b2374e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:31.850299 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.850212    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3a4495e-2010-4901-8c4b-f495f8b2374e-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:31.876699 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.872919    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3a4495e-2010-4901-8c4b-f495f8b2374e-kube-api-access-nvz9f" pod "b3a4495e-2010-4901-8c4b-f495f8b2374e" (UID: "b3a4495e-2010-4901-8c4b-f495f8b2374e"). InnerVolumeSpecName "kube-api-access-nvz9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:31.896699 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.896675    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f"
Oct 30 22:25:31.951594 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.951560    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aa8b895-0530-439f-b390-042fe1c36f28-kube-api-access-r4j5h\" (UniqueName: \"kubernetes.io/projected/2aa8b895-0530-439f-b390-042fe1c36f28-kube-api-access-r4j5h\") pod \"2aa8b895-0530-439f-b390-042fe1c36f28\" (UID: \"2aa8b895-0530-439f-b390-042fe1c36f28\") "
Oct 30 22:25:31.951838 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.951768    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aa8b895-0530-439f-b390-042fe1c36f28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aa8b895-0530-439f-b390-042fe1c36f28-plugins-config-volume\") pod \"2aa8b895-0530-439f-b390-042fe1c36f28\" (UID: \"2aa8b895-0530-439f-b390-042fe1c36f28\") "
Oct 30 22:25:31.952579 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.952528    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvz9f\" (UniqueName: \"kubernetes.io/projected/b3a4495e-2010-4901-8c4b-f495f8b2374e-kube-api-access-nvz9f\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:31.952579 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.952529    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aa8b895-0530-439f-b390-042fe1c36f28-plugins-config-volume" pod "2aa8b895-0530-439f-b390-042fe1c36f28" (UID: "2aa8b895-0530-439f-b390-042fe1c36f28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:31.964814 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:31.964780    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa8b895-0530-439f-b390-042fe1c36f28-kube-api-access-r4j5h" pod "2aa8b895-0530-439f-b390-042fe1c36f28" (UID: "2aa8b895-0530-439f-b390-042fe1c36f28"). InnerVolumeSpecName "kube-api-access-r4j5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:32.054272 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.054152    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aa8b895-0530-439f-b390-042fe1c36f28-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:32.054272 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.054181    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4j5h\" (UniqueName: \"kubernetes.io/projected/2aa8b895-0530-439f-b390-042fe1c36f28-kube-api-access-r4j5h\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:32.068271 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.068240    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"]
Oct 30 22:25:32.068530 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:25:32.068283    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c75d3c_bba8_4a52_a0cf_f76a2b4b8495.slice/crio-30999e8beeb06668d7083ecf85e3708998c34d8d9a82d01f0f0e626e18224bf0 WatchSource:0}: Error finding container 30999e8beeb06668d7083ecf85e3708998c34d8d9a82d01f0f0e626e18224bf0: Status 404 returned error can't find the container with id 30999e8beeb06668d7083ecf85e3708998c34d8d9a82d01f0f0e626e18224bf0
Oct 30 22:25:32.070906 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.070873    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:25:32.071033 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.070940    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:25:32.071033 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.070969    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:25:32.258155 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.257964    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:32.277400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.276484    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn" event={"ID":"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495","Type":"ContainerStarted","Data":"8595aa86978856d85bb6582032e10dcd5561f2ba65c802c5800c469f310a76bd"}
Oct 30 22:25:32.277400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.276544    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn" event={"ID":"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495","Type":"ContainerStarted","Data":"30999e8beeb06668d7083ecf85e3708998c34d8d9a82d01f0f0e626e18224bf0"}
Oct 30 22:25:32.282254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.278208    4889 generic.go:356] "Generic (PLEG): container finished" podID="2aa8b895-0530-439f-b390-042fe1c36f28" containerID="3c79fa09a8c8830c8672623d0e9b559254cc1f1e12b939d36f98720f13db464b" exitCode=0
Oct 30 22:25:32.282254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.278347    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f" event={"ID":"2aa8b895-0530-439f-b390-042fe1c36f28","Type":"ContainerDied","Data":"3c79fa09a8c8830c8672623d0e9b559254cc1f1e12b939d36f98720f13db464b"}
Oct 30 22:25:32.282254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.278389    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f" event={"ID":"2aa8b895-0530-439f-b390-042fe1c36f28","Type":"ContainerDied","Data":"263ada31897eb2010eb9aba2318f1e6881bb39e8987614a66e8dd86f0cf4749a"}
Oct 30 22:25:32.282254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.278417    4889 scope.go:122] "RemoveContainer" containerID="3c79fa09a8c8830c8672623d0e9b559254cc1f1e12b939d36f98720f13db464b"
Oct 30 22:25:32.282254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.278560    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f"
Oct 30 22:25:32.282953 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.282932    4889 generic.go:356] "Generic (PLEG): container finished" podID="b3a4495e-2010-4901-8c4b-f495f8b2374e" containerID="158274f9bbc1eb68357b750e021b7dcd65ca48b16b89ed519c3ed4191a04dc83" exitCode=0
Oct 30 22:25:32.283153 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.282963    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8"
Oct 30 22:25:32.283345 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.282977    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8" event={"ID":"b3a4495e-2010-4901-8c4b-f495f8b2374e","Type":"ContainerDied","Data":"158274f9bbc1eb68357b750e021b7dcd65ca48b16b89ed519c3ed4191a04dc83"}
Oct 30 22:25:32.283489 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.283474    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8" event={"ID":"b3a4495e-2010-4901-8c4b-f495f8b2374e","Type":"ContainerDied","Data":"24ac3e13a584cfaf3b94be05897f29b5a8a96e4f448ce552769a69ce84b2513d"}
Oct 30 22:25:32.300177 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.300100    4889 scope.go:122] "RemoveContainer" containerID="3c79fa09a8c8830c8672623d0e9b559254cc1f1e12b939d36f98720f13db464b"
Oct 30 22:25:32.300469 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:25:32.300442    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c79fa09a8c8830c8672623d0e9b559254cc1f1e12b939d36f98720f13db464b\": container with ID starting with 3c79fa09a8c8830c8672623d0e9b559254cc1f1e12b939d36f98720f13db464b not found: ID does not exist" containerID="3c79fa09a8c8830c8672623d0e9b559254cc1f1e12b939d36f98720f13db464b"
Oct 30 22:25:32.300551 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.300479    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c79fa09a8c8830c8672623d0e9b559254cc1f1e12b939d36f98720f13db464b"} err="failed to get container status \"3c79fa09a8c8830c8672623d0e9b559254cc1f1e12b939d36f98720f13db464b\": rpc error: code = NotFound desc = could not find container \"3c79fa09a8c8830c8672623d0e9b559254cc1f1e12b939d36f98720f13db464b\": container with ID starting with 3c79fa09a8c8830c8672623d0e9b559254cc1f1e12b939d36f98720f13db464b not found: ID does not exist"
Oct 30 22:25:32.300551 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.300500    4889 scope.go:122] "RemoveContainer" containerID="158274f9bbc1eb68357b750e021b7dcd65ca48b16b89ed519c3ed4191a04dc83"
Oct 30 22:25:32.315071 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.315020    4889 scope.go:122] "RemoveContainer" containerID="158274f9bbc1eb68357b750e021b7dcd65ca48b16b89ed519c3ed4191a04dc83"
Oct 30 22:25:32.315833 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:25:32.315801    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"158274f9bbc1eb68357b750e021b7dcd65ca48b16b89ed519c3ed4191a04dc83\": container with ID starting with 158274f9bbc1eb68357b750e021b7dcd65ca48b16b89ed519c3ed4191a04dc83 not found: ID does not exist" containerID="158274f9bbc1eb68357b750e021b7dcd65ca48b16b89ed519c3ed4191a04dc83"
Oct 30 22:25:32.315929 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.315846    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"158274f9bbc1eb68357b750e021b7dcd65ca48b16b89ed519c3ed4191a04dc83"} err="failed to get container status \"158274f9bbc1eb68357b750e021b7dcd65ca48b16b89ed519c3ed4191a04dc83\": rpc error: code = NotFound desc = could not find container \"158274f9bbc1eb68357b750e021b7dcd65ca48b16b89ed519c3ed4191a04dc83\": container with ID starting with 158274f9bbc1eb68357b750e021b7dcd65ca48b16b89ed519c3ed4191a04dc83 not found: ID does not exist"
Oct 30 22:25:32.335302 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.334523    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn" podStartSLOduration=2.33450436 podStartE2EDuration="2.33450436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:32.320620078 +0000 UTC m=+1117.676986394" watchObservedRunningTime="2026-10-30 22:25:32.33450436 +0000 UTC m=+1117.690870675"
Oct 30 22:25:32.363185 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.363143    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f"]
Oct 30 22:25:32.370095 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.370059    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-476e593d-796fb9bdc9-sq29f"]
Oct 30 22:25:32.403619 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.403564    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8"]
Oct 30 22:25:32.407602 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.407577    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-476e593d-6555654587-lljr8"]
Oct 30 22:25:32.544275 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.528342    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:32.832461 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.832426    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aa8b895-0530-439f-b390-042fe1c36f28" path="/var/lib/kubelet/pods/2aa8b895-0530-439f-b390-042fe1c36f28/volumes"
Oct 30 22:25:32.832996 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.832965    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3a4495e-2010-4901-8c4b-f495f8b2374e" path="/var/lib/kubelet/pods/b3a4495e-2010-4901-8c4b-f495f8b2374e/volumes"
Oct 30 22:25:32.942309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:32.942277    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:33.128161 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:33.125741    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:33.210204 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:33.210150    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"]
Oct 30 22:25:33.212370 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:33.210839    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v" podUID="e967d2cf-50fa-4e9e-90b9-d49da1b81da1" containerName="istio-proxy" containerID="cri-o://4910b367e11072bf49e3bd3713ce4a895c12dd1cbda0c5d28756284360e66482" gracePeriod=30
Oct 30 22:25:33.472356 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:33.471152    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:33.562009 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:33.561644    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:33.709040 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:33.708998    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z"]
Oct 30 22:25:33.715214 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:33.715185    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:33.716245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:33.716116    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:25:33.721837 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:33.721809    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242"]
Oct 30 22:25:33.876289 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:33.876191    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:34.098329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.098295    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:34.292247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.291818    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:34.641303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.615031    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"]
Oct 30 22:25:34.641303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.616092    4889 cpu_manager.go:414] "containerMap: removing container" podUID="b3a4495e-2010-4901-8c4b-f495f8b2374e" containerName="payload-pre-processing"
Oct 30 22:25:34.641303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.616112    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3a4495e-2010-4901-8c4b-f495f8b2374e" containerName="payload-pre-processing"
Oct 30 22:25:34.641303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.616130    4889 cpu_manager.go:414] "containerMap: removing container" podUID="2aa8b895-0530-439f-b390-042fe1c36f28" containerName="payload-processing"
Oct 30 22:25:34.641303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.616137    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aa8b895-0530-439f-b390-042fe1c36f28" containerName="payload-processing"
Oct 30 22:25:34.641303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.616366    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3a4495e-2010-4901-8c4b-f495f8b2374e" containerName="payload-pre-processing"
Oct 30 22:25:34.641303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.616384    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aa8b895-0530-439f-b390-042fe1c36f28" containerName="payload-processing"
Oct 30 22:25:34.683249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.670117    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"]
Oct 30 22:25:34.683249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.670271    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:34.728254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.702264    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-80ecb3-openshift-default-dockercfg-g6zvp\""
Oct 30 22:25:34.759294 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.756184    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:34.788694 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.788656    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9833755d-9e30-41f6-806f-688a9cb72349-istio-token\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:34.790245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.788935    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-istio-data\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:34.790245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.789044    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-credential-socket\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:34.790245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.789086    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-istio-envoy\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:34.790245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.789117    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9833755d-9e30-41f6-806f-688a9cb72349-istiod-ca-cert\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:34.790245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.789167    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-workload-socket\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:34.790245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.789261    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9833755d-9e30-41f6-806f-688a9cb72349-istio-podinfo\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:34.790245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.789323    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-workload-certs\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:34.790245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.789503    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cclv\" (UniqueName: \"kubernetes.io/projected/9833755d-9e30-41f6-806f-688a9cb72349-kube-api-access-4cclv\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:34.901316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.891000    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cclv\" (UniqueName: \"kubernetes.io/projected/9833755d-9e30-41f6-806f-688a9cb72349-kube-api-access-4cclv\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:34.901316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.891085    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9833755d-9e30-41f6-806f-688a9cb72349-istio-token\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:34.901316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.891123    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-istio-data\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:34.901316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.891191    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-credential-socket\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:34.901316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.891243    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-istio-envoy\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:34.901316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.891270    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9833755d-9e30-41f6-806f-688a9cb72349-istiod-ca-cert\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:34.901316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.891322    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-workload-socket\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:34.901316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.891384    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9833755d-9e30-41f6-806f-688a9cb72349-istio-podinfo\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:34.901316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.891450    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-workload-certs\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:34.901316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.893277    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-istio-data\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:34.901316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.893510    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-credential-socket\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:34.917178 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.915703    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-istio-envoy\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:34.917178 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.916340    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-workload-certs\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:35.006680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.954142    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9833755d-9e30-41f6-806f-688a9cb72349-istiod-ca-cert\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:35.006680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.954699    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-workload-socket\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:35.006680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.955107    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9833755d-9e30-41f6-806f-688a9cb72349-istio-podinfo\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:35.006680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:34.974011    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cclv\" (UniqueName: \"kubernetes.io/projected/9833755d-9e30-41f6-806f-688a9cb72349-kube-api-access-4cclv\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:35.259568 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:35.233979    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9833755d-9e30-41f6-806f-688a9cb72349-istio-token\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:35.259568 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:35.259138    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:35.275794 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:35.275755    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:35.469202 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:35.469165    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:35.713050 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:35.713008    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:35.788145 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:35.787901    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt"]
Oct 30 22:25:35.788632 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:35.788602    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt" podUID="31ca0794-c411-43dc-bfab-589ba56c79cf" containerName="maas-api" containerID="cri-o://8d248ccb2e8f98da7c060363f504baf32a7ac808827acbd850bc7358f210500b" gracePeriod=30
Oct 30 22:25:35.941629 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:35.940544    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"]
Oct 30 22:25:35.952688 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:35.952639    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:25:35.952804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:35.952776    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:25:35.952886 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:35.952811    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:25:35.968758 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:35.968729    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:36.377801 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:36.371671    4889 generic.go:356] "Generic (PLEG): container finished" podID="31ca0794-c411-43dc-bfab-589ba56c79cf" containerID="8d248ccb2e8f98da7c060363f504baf32a7ac808827acbd850bc7358f210500b" exitCode=0
Oct 30 22:25:36.377801 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:36.371886    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt" event={"ID":"31ca0794-c411-43dc-bfab-589ba56c79cf","Type":"ContainerDied","Data":"8d248ccb2e8f98da7c060363f504baf32a7ac808827acbd850bc7358f210500b"}
Oct 30 22:25:36.378018 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:36.377884    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt"
Oct 30 22:25:36.379070 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:36.378984    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n" event={"ID":"9833755d-9e30-41f6-806f-688a9cb72349","Type":"ContainerStarted","Data":"da0e6ca092e95570b9a393d80f2b5144ef5056dd31dbe09c02b06910de65c043"}
Oct 30 22:25:36.379070 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:36.379031    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n" event={"ID":"9833755d-9e30-41f6-806f-688a9cb72349","Type":"ContainerStarted","Data":"b2dcd0628fcb41be4bcb6ecba25d1ea488c8e089116dae001e409bc4c2587846"}
Oct 30 22:25:36.467252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:36.459346    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31ca0794-c411-43dc-bfab-589ba56c79cf-kube-api-access-wjqdr\" (UniqueName: \"kubernetes.io/projected/31ca0794-c411-43dc-bfab-589ba56c79cf-kube-api-access-wjqdr\") pod \"31ca0794-c411-43dc-bfab-589ba56c79cf\" (UID: \"31ca0794-c411-43dc-bfab-589ba56c79cf\") "
Oct 30 22:25:36.467252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:36.459423    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31ca0794-c411-43dc-bfab-589ba56c79cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/31ca0794-c411-43dc-bfab-589ba56c79cf-metrics-certs\") pod \"31ca0794-c411-43dc-bfab-589ba56c79cf\" (UID: \"31ca0794-c411-43dc-bfab-589ba56c79cf\") "
Oct 30 22:25:36.467252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:36.459480    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31ca0794-c411-43dc-bfab-589ba56c79cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ca0794-c411-43dc-bfab-589ba56c79cf-openshift-service-ca\") pod \"31ca0794-c411-43dc-bfab-589ba56c79cf\" (UID: \"31ca0794-c411-43dc-bfab-589ba56c79cf\") "
Oct 30 22:25:36.467252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:36.459517    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31ca0794-c411-43dc-bfab-589ba56c79cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31ca0794-c411-43dc-bfab-589ba56c79cf-maas-api-tls\") pod \"31ca0794-c411-43dc-bfab-589ba56c79cf\" (UID: \"31ca0794-c411-43dc-bfab-589ba56c79cf\") "
Oct 30 22:25:36.481441 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:36.475018    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ca0794-c411-43dc-bfab-589ba56c79cf-kube-api-access-wjqdr" pod "31ca0794-c411-43dc-bfab-589ba56c79cf" (UID: "31ca0794-c411-43dc-bfab-589ba56c79cf"). InnerVolumeSpecName "kube-api-access-wjqdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:36.481441 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:36.476642    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ca0794-c411-43dc-bfab-589ba56c79cf-openshift-service-ca" pod "31ca0794-c411-43dc-bfab-589ba56c79cf" (UID: "31ca0794-c411-43dc-bfab-589ba56c79cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:36.483491 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:36.483374    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ca0794-c411-43dc-bfab-589ba56c79cf-metrics-certs" pod "31ca0794-c411-43dc-bfab-589ba56c79cf" (UID: "31ca0794-c411-43dc-bfab-589ba56c79cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:36.523390 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:36.486262    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n" podStartSLOduration=2.486243302 podStartE2EDuration="2.486243302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:36.483866361 +0000 UTC m=+1121.840232677" watchObservedRunningTime="2026-10-30 22:25:36.486243302 +0000 UTC m=+1121.842609646"
Oct 30 22:25:36.588034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:36.587268    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjqdr\" (UniqueName: \"kubernetes.io/projected/31ca0794-c411-43dc-bfab-589ba56c79cf-kube-api-access-wjqdr\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:36.588034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:36.587308    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31ca0794-c411-43dc-bfab-589ba56c79cf-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:36.588034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:36.587322    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ca0794-c411-43dc-bfab-589ba56c79cf-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:36.640679 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:36.640555    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ca0794-c411-43dc-bfab-589ba56c79cf-maas-api-tls" pod "31ca0794-c411-43dc-bfab-589ba56c79cf" (UID: "31ca0794-c411-43dc-bfab-589ba56c79cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:36.844622 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:36.844579    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31ca0794-c411-43dc-bfab-589ba56c79cf-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:37.243327 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.243294    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:37.276045 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.276005    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:37.282492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.282449    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:37.361336 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.361302    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt"]
Oct 30 22:25:37.362144 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.362122    4889 cpu_manager.go:414] "containerMap: removing container" podUID="31ca0794-c411-43dc-bfab-589ba56c79cf" containerName="maas-api"
Oct 30 22:25:37.362144 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.362143    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ca0794-c411-43dc-bfab-589ba56c79cf" containerName="maas-api"
Oct 30 22:25:37.362419 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.362378    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="31ca0794-c411-43dc-bfab-589ba56c79cf" containerName="maas-api"
Oct 30 22:25:37.365637 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.365610    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt"
Oct 30 22:25:37.383416 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.383382    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2949c143\""
Oct 30 22:25:37.384427 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.384402    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2949c143-dockercfg-j9wfk\""
Oct 30 22:25:37.390018 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.389994    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt"]
Oct 30 22:25:37.520498 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.506123    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8fmh\" (UniqueName: \"kubernetes.io/projected/800dedce-28ae-4996-ac41-0253054aea5a-kube-api-access-l8fmh\") pod \"payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt\" (UID: \"800dedce-28ae-4996-ac41-0253054aea5a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt"
Oct 30 22:25:37.520498 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.506363    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/800dedce-28ae-4996-ac41-0253054aea5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt\" (UID: \"800dedce-28ae-4996-ac41-0253054aea5a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt"
Oct 30 22:25:37.627099 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.627066    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/800dedce-28ae-4996-ac41-0253054aea5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt\" (UID: \"800dedce-28ae-4996-ac41-0253054aea5a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt"
Oct 30 22:25:37.657256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.647502    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/800dedce-28ae-4996-ac41-0253054aea5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt\" (UID: \"800dedce-28ae-4996-ac41-0253054aea5a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt"
Oct 30 22:25:37.657497 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.647537    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7"]
Oct 30 22:25:37.675564 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.675528    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8fmh\" (UniqueName: \"kubernetes.io/projected/800dedce-28ae-4996-ac41-0253054aea5a-kube-api-access-l8fmh\") pod \"payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt\" (UID: \"800dedce-28ae-4996-ac41-0253054aea5a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt"
Oct 30 22:25:37.677035 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.676861    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:37.714576 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.713418    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt"
Oct 30 22:25:37.771307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.769869    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:37.771307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.769927    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt" event={"ID":"31ca0794-c411-43dc-bfab-589ba56c79cf","Type":"ContainerDied","Data":"2083f1bd3ab76129befc0a10a25b6469e33afd454fa900a6d46006720f541c52"}
Oct 30 22:25:37.771307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.770033    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:25:37.771307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.770394    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7"
Oct 30 22:25:37.777449 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.776958    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7"]
Oct 30 22:25:37.861641 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.861579    4889 scope.go:122] "RemoveContainer" containerID="8d248ccb2e8f98da7c060363f504baf32a7ac808827acbd850bc7358f210500b"
Oct 30 22:25:37.871303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.870240    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8fmh\" (UniqueName: \"kubernetes.io/projected/800dedce-28ae-4996-ac41-0253054aea5a-kube-api-access-l8fmh\") pod \"payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt\" (UID: \"800dedce-28ae-4996-ac41-0253054aea5a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt"
Oct 30 22:25:37.959091 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.959061    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:37.968444 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.968412    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6htrf\" (UniqueName: \"kubernetes.io/projected/e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29-kube-api-access-6htrf\") pod \"payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7\" (UID: \"e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29\") " pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7"
Oct 30 22:25:37.968572 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.968522    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29-plugins-config-volume\") pod \"payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7\" (UID: \"e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29\") " pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7"
Oct 30 22:25:37.991626 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:37.991594    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt"
Oct 30 22:25:38.005645 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:38.005609    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt"]
Oct 30 22:25:38.014606 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:38.014573    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-913eddcb-5fccb89bd6-qx4bt"]
Oct 30 22:25:38.041668 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:38.041592    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"]
Oct 30 22:25:38.042130 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:38.042098    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz" podUID="f1c26fbd-ffd1-438b-8c6a-36b667be335c" containerName="istio-proxy" containerID="cri-o://cd0aac5fcff56bd903844172a57158ecb1d1e322ce3189f3cc2da4dc1ceebeac" gracePeriod=30
Oct 30 22:25:38.070401 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:38.069888    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6htrf\" (UniqueName: \"kubernetes.io/projected/e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29-kube-api-access-6htrf\") pod \"payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7\" (UID: \"e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29\") " pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7"
Oct 30 22:25:38.070401 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:38.070001    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29-plugins-config-volume\") pod \"payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7\" (UID: \"e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29\") " pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7"
Oct 30 22:25:38.072202 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:38.072163    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29-plugins-config-volume\") pod \"payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7\" (UID: \"e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29\") " pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7"
Oct 30 22:25:38.101372 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:38.100622    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6htrf\" (UniqueName: \"kubernetes.io/projected/e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29-kube-api-access-6htrf\") pod \"payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7\" (UID: \"e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29\") " pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7"
Oct 30 22:25:38.135653 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:38.135431    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:38.218559 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:38.217904    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7"
Oct 30 22:25:38.379456 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:38.379364    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt"]
Oct 30 22:25:38.505771 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:25:38.505730    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod800dedce_28ae_4996_ac41_0253054aea5a.slice/crio-83c3745c1756026f9d5e8511559d22154754ed80eeafd7c2afee564c553986e5 WatchSource:0}: Error finding container 83c3745c1756026f9d5e8511559d22154754ed80eeafd7c2afee564c553986e5: Status 404 returned error can't find the container with id 83c3745c1756026f9d5e8511559d22154754ed80eeafd7c2afee564c553986e5
Oct 30 22:25:38.710694 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:38.710510    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z" podUID="0f1b6c42-a7aa-468b-9974-09e2e80338c5" containerName="payload-processing" containerID="cri-o://ea05da12c1b9d68fdd78055c8ca711de91f20d03cf6076717f143f23112a9a65" gracePeriod=25
Oct 30 22:25:38.722917 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:38.722869    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242" podUID="e7b8353e-019c-4ebf-9a34-771262f281e9" containerName="payload-pre-processing" containerID="cri-o://55076776392fa44c52d98346f2192799b98066b22b75fea591bd01168a533a47" gracePeriod=25
Oct 30 22:25:38.727695 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:38.727322    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt" event={"ID":"800dedce-28ae-4996-ac41-0253054aea5a","Type":"ContainerStarted","Data":"83c3745c1756026f9d5e8511559d22154754ed80eeafd7c2afee564c553986e5"}
Oct 30 22:25:38.780658 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:38.780616    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31ca0794-c411-43dc-bfab-589ba56c79cf" path="/var/lib/kubelet/pods/31ca0794-c411-43dc-bfab-589ba56c79cf/volumes"
Oct 30 22:25:38.965168 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:38.952884    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7"]
Oct 30 22:25:38.973726 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:38.973530    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:38.975555 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:25:38.975503    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e63a57_6ca5_4ce2_8843_8cf9f41e3c29.slice/crio-0732acc34f5561468113cf64834a4497995105058754e0f9e3d26397a886aa6b WatchSource:0}: Error finding container 0732acc34f5561468113cf64834a4497995105058754e0f9e3d26397a886aa6b: Status 404 returned error can't find the container with id 0732acc34f5561468113cf64834a4497995105058754e0f9e3d26397a886aa6b
Oct 30 22:25:39.132202 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.132172    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z"
Oct 30 22:25:39.141868 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.141816    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242"
Oct 30 22:25:39.143300 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.143269    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:39.225287 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.224293    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7b8353e-019c-4ebf-9a34-771262f281e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7b8353e-019c-4ebf-9a34-771262f281e9-plugins-config-volume\") pod \"e7b8353e-019c-4ebf-9a34-771262f281e9\" (UID: \"e7b8353e-019c-4ebf-9a34-771262f281e9\") "
Oct 30 22:25:39.225287 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.224418    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f1b6c42-a7aa-468b-9974-09e2e80338c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1b6c42-a7aa-468b-9974-09e2e80338c5-plugins-config-volume\") pod \"0f1b6c42-a7aa-468b-9974-09e2e80338c5\" (UID: \"0f1b6c42-a7aa-468b-9974-09e2e80338c5\") "
Oct 30 22:25:39.225287 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.224579    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f1b6c42-a7aa-468b-9974-09e2e80338c5-kube-api-access-jn5jt\" (UniqueName: \"kubernetes.io/projected/0f1b6c42-a7aa-468b-9974-09e2e80338c5-kube-api-access-jn5jt\") pod \"0f1b6c42-a7aa-468b-9974-09e2e80338c5\" (UID: \"0f1b6c42-a7aa-468b-9974-09e2e80338c5\") "
Oct 30 22:25:39.225287 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.224628    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7b8353e-019c-4ebf-9a34-771262f281e9-kube-api-access-wspg5\" (UniqueName: \"kubernetes.io/projected/e7b8353e-019c-4ebf-9a34-771262f281e9-kube-api-access-wspg5\") pod \"e7b8353e-019c-4ebf-9a34-771262f281e9\" (UID: \"e7b8353e-019c-4ebf-9a34-771262f281e9\") "
Oct 30 22:25:39.230293 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.230258    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f1b6c42-a7aa-468b-9974-09e2e80338c5-plugins-config-volume" pod "0f1b6c42-a7aa-468b-9974-09e2e80338c5" (UID: "0f1b6c42-a7aa-468b-9974-09e2e80338c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:39.230748 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.230721    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7b8353e-019c-4ebf-9a34-771262f281e9-plugins-config-volume" pod "e7b8353e-019c-4ebf-9a34-771262f281e9" (UID: "e7b8353e-019c-4ebf-9a34-771262f281e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:39.238965 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.238920    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f1b6c42-a7aa-468b-9974-09e2e80338c5-kube-api-access-jn5jt" pod "0f1b6c42-a7aa-468b-9974-09e2e80338c5" (UID: "0f1b6c42-a7aa-468b-9974-09e2e80338c5"). InnerVolumeSpecName "kube-api-access-jn5jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:39.241045 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.241012    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7b8353e-019c-4ebf-9a34-771262f281e9-kube-api-access-wspg5" pod "e7b8353e-019c-4ebf-9a34-771262f281e9" (UID: "e7b8353e-019c-4ebf-9a34-771262f281e9"). InnerVolumeSpecName "kube-api-access-wspg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:39.359032 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.354313    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7b8353e-019c-4ebf-9a34-771262f281e9-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:39.359032 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.354356    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1b6c42-a7aa-468b-9974-09e2e80338c5-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:39.359032 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.354374    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn5jt\" (UniqueName: \"kubernetes.io/projected/0f1b6c42-a7aa-468b-9974-09e2e80338c5-kube-api-access-jn5jt\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:39.359032 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.354388    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wspg5\" (UniqueName: \"kubernetes.io/projected/e7b8353e-019c-4ebf-9a34-771262f281e9-kube-api-access-wspg5\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:39.741673 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.741592    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt" event={"ID":"800dedce-28ae-4996-ac41-0253054aea5a","Type":"ContainerStarted","Data":"138e6e5f9bfb614a72240554aad30eec86b260229fabb74e65a384cb54098662"}
Oct 30 22:25:39.742255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.742202    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt"
Oct 30 22:25:39.748131 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.748096    4889 generic.go:356] "Generic (PLEG): container finished" podID="0f1b6c42-a7aa-468b-9974-09e2e80338c5" containerID="ea05da12c1b9d68fdd78055c8ca711de91f20d03cf6076717f143f23112a9a65" exitCode=0
Oct 30 22:25:39.748287 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.748143    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z"
Oct 30 22:25:39.748287 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.748167    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z" event={"ID":"0f1b6c42-a7aa-468b-9974-09e2e80338c5","Type":"ContainerDied","Data":"ea05da12c1b9d68fdd78055c8ca711de91f20d03cf6076717f143f23112a9a65"}
Oct 30 22:25:39.748287 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.748209    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z" event={"ID":"0f1b6c42-a7aa-468b-9974-09e2e80338c5","Type":"ContainerDied","Data":"d1939907f74584c4feac72ece45e566b6025b74483707dca53a40b900d53e0d3"}
Oct 30 22:25:39.748287 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.748279    4889 scope.go:122] "RemoveContainer" containerID="ea05da12c1b9d68fdd78055c8ca711de91f20d03cf6076717f143f23112a9a65"
Oct 30 22:25:39.748489 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.748450    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"]
Oct 30 22:25:39.749575 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.749353    4889 cpu_manager.go:414] "containerMap: removing container" podUID="0f1b6c42-a7aa-468b-9974-09e2e80338c5" containerName="payload-processing"
Oct 30 22:25:39.749575 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.749371    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f1b6c42-a7aa-468b-9974-09e2e80338c5" containerName="payload-processing"
Oct 30 22:25:39.749575 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.749472    4889 cpu_manager.go:414] "containerMap: removing container" podUID="e7b8353e-019c-4ebf-9a34-771262f281e9" containerName="payload-pre-processing"
Oct 30 22:25:39.749575 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.749482    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7b8353e-019c-4ebf-9a34-771262f281e9" containerName="payload-pre-processing"
Oct 30 22:25:39.749701 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.749633    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7b8353e-019c-4ebf-9a34-771262f281e9" containerName="payload-pre-processing"
Oct 30 22:25:39.749701 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.749649    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f1b6c42-a7aa-468b-9974-09e2e80338c5" containerName="payload-processing"
Oct 30 22:25:39.753348 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.750871    4889 generic.go:356] "Generic (PLEG): container finished" podID="e7b8353e-019c-4ebf-9a34-771262f281e9" containerID="55076776392fa44c52d98346f2192799b98066b22b75fea591bd01168a533a47" exitCode=0
Oct 30 22:25:39.753348 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.751019    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242"
Oct 30 22:25:39.761515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.760828    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242" event={"ID":"e7b8353e-019c-4ebf-9a34-771262f281e9","Type":"ContainerDied","Data":"55076776392fa44c52d98346f2192799b98066b22b75fea591bd01168a533a47"}
Oct 30 22:25:39.761515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.760865    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242" event={"ID":"e7b8353e-019c-4ebf-9a34-771262f281e9","Type":"ContainerDied","Data":"b7a9ac614e698d74234d87f385b477a84b623625c29a435712b9749987304b86"}
Oct 30 22:25:39.761515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.760938    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:39.766591 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.766380    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7" event={"ID":"e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29","Type":"ContainerStarted","Data":"81a56a08a6c26da7b262052180d5c3babebb44e8929d746683018382d74ea536"}
Oct 30 22:25:39.766591 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.766420    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7"
Oct 30 22:25:39.766591 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.766436    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7" event={"ID":"e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29","Type":"ContainerStarted","Data":"0732acc34f5561468113cf64834a4497995105058754e0f9e3d26397a886aa6b"}
Oct 30 22:25:39.766591 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.766502    4889 scope.go:122] "RemoveContainer" containerID="ea05da12c1b9d68fdd78055c8ca711de91f20d03cf6076717f143f23112a9a65"
Oct 30 22:25:39.779743 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:25:39.767833    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea05da12c1b9d68fdd78055c8ca711de91f20d03cf6076717f143f23112a9a65\": container with ID starting with ea05da12c1b9d68fdd78055c8ca711de91f20d03cf6076717f143f23112a9a65 not found: ID does not exist" containerID="ea05da12c1b9d68fdd78055c8ca711de91f20d03cf6076717f143f23112a9a65"
Oct 30 22:25:39.779743 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.767886    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea05da12c1b9d68fdd78055c8ca711de91f20d03cf6076717f143f23112a9a65"} err="failed to get container status \"ea05da12c1b9d68fdd78055c8ca711de91f20d03cf6076717f143f23112a9a65\": rpc error: code = NotFound desc = could not find container \"ea05da12c1b9d68fdd78055c8ca711de91f20d03cf6076717f143f23112a9a65\": container with ID starting with ea05da12c1b9d68fdd78055c8ca711de91f20d03cf6076717f143f23112a9a65 not found: ID does not exist"
Oct 30 22:25:39.779743 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.767929    4889 scope.go:122] "RemoveContainer" containerID="55076776392fa44c52d98346f2192799b98066b22b75fea591bd01168a533a47"
Oct 30 22:25:39.779743 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.777816    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"]
Oct 30 22:25:39.779743 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.779550    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-5983ea-openshift-default-dockercfg-mtrdw\""
Oct 30 22:25:39.783087 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.783067    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:39.812982 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.811137    4889 scope.go:122] "RemoveContainer" containerID="55076776392fa44c52d98346f2192799b98066b22b75fea591bd01168a533a47"
Oct 30 22:25:39.812982 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:25:39.811596    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55076776392fa44c52d98346f2192799b98066b22b75fea591bd01168a533a47\": container with ID starting with 55076776392fa44c52d98346f2192799b98066b22b75fea591bd01168a533a47 not found: ID does not exist" containerID="55076776392fa44c52d98346f2192799b98066b22b75fea591bd01168a533a47"
Oct 30 22:25:39.812982 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.811636    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55076776392fa44c52d98346f2192799b98066b22b75fea591bd01168a533a47"} err="failed to get container status \"55076776392fa44c52d98346f2192799b98066b22b75fea591bd01168a533a47\": rpc error: code = NotFound desc = could not find container \"55076776392fa44c52d98346f2192799b98066b22b75fea591bd01168a533a47\": container with ID starting with 55076776392fa44c52d98346f2192799b98066b22b75fea591bd01168a533a47 not found: ID does not exist"
Oct 30 22:25:39.812982 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.812693    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt" podStartSLOduration=2.812677431 podStartE2EDuration="2.812677431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:39.790309963 +0000 UTC m=+1125.146676278" watchObservedRunningTime="2026-10-30 22:25:39.812677431 +0000 UTC m=+1125.169043745"
Oct 30 22:25:39.882103 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.882069    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-envoy\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:39.882325 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.882303    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-podinfo\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:39.882397 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.882368    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-workload-certs\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:39.882597 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.882569    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drj8z\" (UniqueName: \"kubernetes.io/projected/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-kube-api-access-drj8z\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:39.882696 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.882663    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istiod-ca-cert\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:39.882749 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.882703    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-workload-socket\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:39.882810 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.882767    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-credential-socket\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:39.882975 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.882938    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-token\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:39.883109 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.883087    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-data\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:39.891644 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.891610    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z"]
Oct 30 22:25:39.895899 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.895865    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-913eddcb-d7477fc45-nd68z"]
Oct 30 22:25:39.967750 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.967692    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:39.989630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.987542    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-token\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:39.989630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.987651    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-data\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:39.989630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.987703    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-envoy\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:39.989630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.987792    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-podinfo\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:39.989630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.987862    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-workload-certs\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:39.989630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.988012    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drj8z\" (UniqueName: \"kubernetes.io/projected/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-kube-api-access-drj8z\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:39.989630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.988096    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istiod-ca-cert\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:39.989630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.988131    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-workload-socket\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:39.989630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.988178    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-credential-socket\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:39.989630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.988724    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-credential-socket\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:39.989630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:39.989317    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-data\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:40.002816 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.002683    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7" podStartSLOduration=3.00266196 podStartE2EDuration="3.00266196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:39.954457518 +0000 UTC m=+1125.310823834" watchObservedRunningTime="2026-10-30 22:25:40.00266196 +0000 UTC m=+1125.359028274"
Oct 30 22:25:40.005909 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.005089    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-workload-certs\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:40.005909 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.005537    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-workload-socket\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:40.005909 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.005814    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istiod-ca-cert\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:40.047965 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.042876    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-token\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:40.047965 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.042947    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242"]
Oct 30 22:25:40.081265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.080579    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-envoy\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:40.081715 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.081665    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drj8z\" (UniqueName: \"kubernetes.io/projected/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-kube-api-access-drj8z\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:40.093217 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.093142    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-913eddcb-865ff6c5f5-lc242"]
Oct 30 22:25:40.110778 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.106527    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-podinfo\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:40.163796 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.163752    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:40.202085 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.202049    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:40.396372 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:25:40.396334    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:25:40.732861 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.689205    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"]
Oct 30 22:25:40.732861 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.694444    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:25:40.732861 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.694503    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:25:40.732861 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.694543    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:25:40.783812 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.783148    4889 patch_prober.go:29] interesting pod/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": read tcp 10.132.0.2:35094->10.132.0.91:15021: read: connection reset by peer" start-of-body=
Oct 30 22:25:40.783812 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.783200    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v" podUID="e967d2cf-50fa-4e9e-90b9-d49da1b81da1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": read tcp 10.132.0.2:35094->10.132.0.91:15021: read: connection reset by peer"
Oct 30 22:25:40.785936 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.785854    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f1b6c42-a7aa-468b-9974-09e2e80338c5" path="/var/lib/kubelet/pods/0f1b6c42-a7aa-468b-9974-09e2e80338c5/volumes"
Oct 30 22:25:40.786871 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.786819    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7b8353e-019c-4ebf-9a34-771262f281e9" path="/var/lib/kubelet/pods/e7b8353e-019c-4ebf-9a34-771262f281e9/volumes"
Oct 30 22:25:40.787640 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.787596    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg" event={"ID":"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab","Type":"ContainerStarted","Data":"acee3959f7a8ff1c1b5e6f4e56a3c97046c0955298ab9883b0a770de90546c16"}
Oct 30 22:25:40.787749 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.787673    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg" event={"ID":"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab","Type":"ContainerStarted","Data":"01d8938e3ee17f4d72333e7c5c4fb075fb040ad250d91c03889304a9633ad398"}
Oct 30 22:25:40.816452 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.816415    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:40.835329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.835282    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg" podStartSLOduration=1.835268181 podStartE2EDuration="1.835268181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:40.83259772 +0000 UTC m=+1126.188964060" watchObservedRunningTime="2026-10-30 22:25:40.835268181 +0000 UTC m=+1126.191634498"
Oct 30 22:25:40.931425 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.931308    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx"]
Oct 30 22:25:40.934908 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.934885    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx"
Oct 30 22:25:40.958074 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.954056    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-80ecb3-dockercfg-jbttq\""
Oct 30 22:25:40.961609 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.961576    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx"]
Oct 30 22:25:40.986770 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:40.986736    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-80ecb3\""
Oct 30 22:25:41.014247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.010981    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:41.082271 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.074034    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0df870-2a4a-404b-9678-bdc054dce3e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx\" (UID: \"cc0df870-2a4a-404b-9678-bdc054dce3e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx"
Oct 30 22:25:41.082271 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.074082    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lt56\" (UniqueName: \"kubernetes.io/projected/cc0df870-2a4a-404b-9678-bdc054dce3e3-kube-api-access-6lt56\") pod \"payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx\" (UID: \"cc0df870-2a4a-404b-9678-bdc054dce3e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx"
Oct 30 22:25:41.176255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.172419    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:41.193468 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.177773    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0df870-2a4a-404b-9678-bdc054dce3e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx\" (UID: \"cc0df870-2a4a-404b-9678-bdc054dce3e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx"
Oct 30 22:25:41.193741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.193712    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lt56\" (UniqueName: \"kubernetes.io/projected/cc0df870-2a4a-404b-9678-bdc054dce3e3-kube-api-access-6lt56\") pod \"payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx\" (UID: \"cc0df870-2a4a-404b-9678-bdc054dce3e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx"
Oct 30 22:25:41.207317 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.195294    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:25:41.207317 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.195345    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg" podUID="c02298a7-464d-4cc4-8bef-3be1dfc6f3ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 22:25:41.207317 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.196557    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0df870-2a4a-404b-9678-bdc054dce3e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx\" (UID: \"cc0df870-2a4a-404b-9678-bdc054dce3e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx"
Oct 30 22:25:41.308666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.261713    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lt56\" (UniqueName: \"kubernetes.io/projected/cc0df870-2a4a-404b-9678-bdc054dce3e3-kube-api-access-6lt56\") pod \"payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx\" (UID: \"cc0df870-2a4a-404b-9678-bdc054dce3e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx"
Oct 30 22:25:41.429251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.405511    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx"
Oct 30 22:25:41.507356 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.507321    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:41.512037 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.512018    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:25:41.535588 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.535531    4889 patch_prober.go:29] interesting pod/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": read tcp 10.132.0.2:41168->10.132.0.92:15021: read: connection reset by peer" start-of-body=
Oct 30 22:25:41.535764 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.535614    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz" podUID="f1c26fbd-ffd1-438b-8c6a-36b667be335c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": read tcp 10.132.0.2:41168->10.132.0.92:15021: read: connection reset by peer"
Oct 30 22:25:41.624857 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.624815    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-data\") pod \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") "
Oct 30 22:25:41.625049 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.625018    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-workload-certs\") pod \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") "
Oct 30 22:25:41.625112 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.625074    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-workload-socket\") pod \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") "
Oct 30 22:25:41.625168 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.625107    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-credential-socket\") pod \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") "
Oct 30 22:25:41.625168 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.625143    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-envoy\") pod \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") "
Oct 30 22:25:41.625349 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.625332    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-kube-api-access-7dnkz\" (UniqueName: \"kubernetes.io/projected/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-kube-api-access-7dnkz\") pod \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") "
Oct 30 22:25:41.625411 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.625383    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-token\" (UniqueName: \"kubernetes.io/projected/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-token\") pod \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") "
Oct 30 22:25:41.625411 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.625381    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-workload-certs" pod "e967d2cf-50fa-4e9e-90b9-d49da1b81da1" (UID: "e967d2cf-50fa-4e9e-90b9-d49da1b81da1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:41.625516 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.625444    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-podinfo\") pod \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") "
Oct 30 22:25:41.625569 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.625539    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istiod-ca-cert\") pod \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\" (UID: \"e967d2cf-50fa-4e9e-90b9-d49da1b81da1\") "
Oct 30 22:25:41.625680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.625655    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-credential-socket" pod "e967d2cf-50fa-4e9e-90b9-d49da1b81da1" (UID: "e967d2cf-50fa-4e9e-90b9-d49da1b81da1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:41.625899 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.625879    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-workload-socket" pod "e967d2cf-50fa-4e9e-90b9-d49da1b81da1" (UID: "e967d2cf-50fa-4e9e-90b9-d49da1b81da1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:41.626726 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.626142    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-data" pod "e967d2cf-50fa-4e9e-90b9-d49da1b81da1" (UID: "e967d2cf-50fa-4e9e-90b9-d49da1b81da1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:41.626726 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.626577    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istiod-ca-cert" pod "e967d2cf-50fa-4e9e-90b9-d49da1b81da1" (UID: "e967d2cf-50fa-4e9e-90b9-d49da1b81da1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:41.626930 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.626884    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:41.626930 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.626904    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:41.626930 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.626919    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:41.627088 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.626934    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:41.627088 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.626949    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:41.629867 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.629830    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-envoy" pod "e967d2cf-50fa-4e9e-90b9-d49da1b81da1" (UID: "e967d2cf-50fa-4e9e-90b9-d49da1b81da1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:41.630417 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.630384    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-podinfo" pod "e967d2cf-50fa-4e9e-90b9-d49da1b81da1" (UID: "e967d2cf-50fa-4e9e-90b9-d49da1b81da1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:25:41.631619 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.631587    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-token" pod "e967d2cf-50fa-4e9e-90b9-d49da1b81da1" (UID: "e967d2cf-50fa-4e9e-90b9-d49da1b81da1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:41.632152 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.632126    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-kube-api-access-7dnkz" pod "e967d2cf-50fa-4e9e-90b9-d49da1b81da1" (UID: "e967d2cf-50fa-4e9e-90b9-d49da1b81da1"). InnerVolumeSpecName "kube-api-access-7dnkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:41.673519 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.673478    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:41.728088 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.728025    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:41.728088 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.728051    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:41.728088 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.728061    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dnkz\" (UniqueName: \"kubernetes.io/projected/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-kube-api-access-7dnkz\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:41.728088 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.728070    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e967d2cf-50fa-4e9e-90b9-d49da1b81da1-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:41.730325 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.730306    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:25:41.797024 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.796992    4889 generic.go:356] "Generic (PLEG): container finished" podID="f1c26fbd-ffd1-438b-8c6a-36b667be335c" containerID="cd0aac5fcff56bd903844172a57158ecb1d1e322ce3189f3cc2da4dc1ceebeac" exitCode=0
Oct 30 22:25:41.797236 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.797053    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"
Oct 30 22:25:41.797236 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.797075    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz" event={"ID":"f1c26fbd-ffd1-438b-8c6a-36b667be335c","Type":"ContainerDied","Data":"cd0aac5fcff56bd903844172a57158ecb1d1e322ce3189f3cc2da4dc1ceebeac"}
Oct 30 22:25:41.797236 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.797133    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz" event={"ID":"f1c26fbd-ffd1-438b-8c6a-36b667be335c","Type":"ContainerDied","Data":"032779cab05dee3f739d3c841dd2b0fb13c93c73bee7c817d23a68639434a672"}
Oct 30 22:25:41.797236 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.797162    4889 scope.go:122] "RemoveContainer" containerID="cd0aac5fcff56bd903844172a57158ecb1d1e322ce3189f3cc2da4dc1ceebeac"
Oct 30 22:25:41.799294 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.799267    4889 generic.go:356] "Generic (PLEG): container finished" podID="e967d2cf-50fa-4e9e-90b9-d49da1b81da1" containerID="4910b367e11072bf49e3bd3713ce4a895c12dd1cbda0c5d28756284360e66482" exitCode=0
Oct 30 22:25:41.799423 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.799317    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v" event={"ID":"e967d2cf-50fa-4e9e-90b9-d49da1b81da1","Type":"ContainerDied","Data":"4910b367e11072bf49e3bd3713ce4a895c12dd1cbda0c5d28756284360e66482"}
Oct 30 22:25:41.799423 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.799364    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v" event={"ID":"e967d2cf-50fa-4e9e-90b9-d49da1b81da1","Type":"ContainerDied","Data":"63f25645a8b1336cdae5fe4a14c8b60282761abfcd39457f19d534c19dbcab2e"}
Oct 30 22:25:41.799612 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.799597    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"
Oct 30 22:25:41.810780 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.810756    4889 scope.go:122] "RemoveContainer" containerID="cd0aac5fcff56bd903844172a57158ecb1d1e322ce3189f3cc2da4dc1ceebeac"
Oct 30 22:25:41.811143 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:25:41.811120    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd0aac5fcff56bd903844172a57158ecb1d1e322ce3189f3cc2da4dc1ceebeac\": container with ID starting with cd0aac5fcff56bd903844172a57158ecb1d1e322ce3189f3cc2da4dc1ceebeac not found: ID does not exist" containerID="cd0aac5fcff56bd903844172a57158ecb1d1e322ce3189f3cc2da4dc1ceebeac"
Oct 30 22:25:41.811244 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.811152    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd0aac5fcff56bd903844172a57158ecb1d1e322ce3189f3cc2da4dc1ceebeac"} err="failed to get container status \"cd0aac5fcff56bd903844172a57158ecb1d1e322ce3189f3cc2da4dc1ceebeac\": rpc error: code = NotFound desc = could not find container \"cd0aac5fcff56bd903844172a57158ecb1d1e322ce3189f3cc2da4dc1ceebeac\": container with ID starting with cd0aac5fcff56bd903844172a57158ecb1d1e322ce3189f3cc2da4dc1ceebeac not found: ID does not exist"
Oct 30 22:25:41.811244 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.811170    4889 scope.go:122] "RemoveContainer" containerID="4910b367e11072bf49e3bd3713ce4a895c12dd1cbda0c5d28756284360e66482"
Oct 30 22:25:41.823843 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.823820    4889 scope.go:122] "RemoveContainer" containerID="4910b367e11072bf49e3bd3713ce4a895c12dd1cbda0c5d28756284360e66482"
Oct 30 22:25:41.824139 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:25:41.824122    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4910b367e11072bf49e3bd3713ce4a895c12dd1cbda0c5d28756284360e66482\": container with ID starting with 4910b367e11072bf49e3bd3713ce4a895c12dd1cbda0c5d28756284360e66482 not found: ID does not exist" containerID="4910b367e11072bf49e3bd3713ce4a895c12dd1cbda0c5d28756284360e66482"
Oct 30 22:25:41.824207 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.824151    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4910b367e11072bf49e3bd3713ce4a895c12dd1cbda0c5d28756284360e66482"} err="failed to get container status \"4910b367e11072bf49e3bd3713ce4a895c12dd1cbda0c5d28756284360e66482\": rpc error: code = NotFound desc = could not find container \"4910b367e11072bf49e3bd3713ce4a895c12dd1cbda0c5d28756284360e66482\": container with ID starting with 4910b367e11072bf49e3bd3713ce4a895c12dd1cbda0c5d28756284360e66482 not found: ID does not exist"
Oct 30 22:25:41.828426 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.828390    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx"]
Oct 30 22:25:41.830501 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.828787    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-token\" (UniqueName: \"kubernetes.io/projected/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-token\") pod \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") "
Oct 30 22:25:41.830501 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.828861    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-credential-socket\") pod \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") "
Oct 30 22:25:41.830501 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.828901    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-envoy\") pod \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") "
Oct 30 22:25:41.830501 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.829047    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-podinfo\") pod \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") "
Oct 30 22:25:41.830501 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.829170    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-credential-socket" pod "f1c26fbd-ffd1-438b-8c6a-36b667be335c" (UID: "f1c26fbd-ffd1-438b-8c6a-36b667be335c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:41.830501 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.829406    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-workload-socket\") pod \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") "
Oct 30 22:25:41.830501 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.829439    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-workload-socket" pod "f1c26fbd-ffd1-438b-8c6a-36b667be335c" (UID: "f1c26fbd-ffd1-438b-8c6a-36b667be335c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:41.830501 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.829522    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-data\") pod \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") "
Oct 30 22:25:41.830501 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.829623    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1c26fbd-ffd1-438b-8c6a-36b667be335c-kube-api-access-fsc9s\" (UniqueName: \"kubernetes.io/projected/f1c26fbd-ffd1-438b-8c6a-36b667be335c-kube-api-access-fsc9s\") pod \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") "
Oct 30 22:25:41.830501 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.829794    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-data" pod "f1c26fbd-ffd1-438b-8c6a-36b667be335c" (UID: "f1c26fbd-ffd1-438b-8c6a-36b667be335c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:41.830501 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.830059    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-workload-certs\") pod \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") "
Oct 30 22:25:41.830501 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.830090    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istiod-ca-cert\") pod \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\" (UID: \"f1c26fbd-ffd1-438b-8c6a-36b667be335c\") "
Oct 30 22:25:41.830501 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.830319    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-workload-certs" pod "f1c26fbd-ffd1-438b-8c6a-36b667be335c" (UID: "f1c26fbd-ffd1-438b-8c6a-36b667be335c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:41.831168 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.830689    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istiod-ca-cert" pod "f1c26fbd-ffd1-438b-8c6a-36b667be335c" (UID: "f1c26fbd-ffd1-438b-8c6a-36b667be335c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:41.831168 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.831157    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:41.831329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.831175    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:41.831329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.831188    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:41.831329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.831199    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:41.831329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.831210    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:41.831720 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.831696    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-envoy" pod "f1c26fbd-ffd1-438b-8c6a-36b667be335c" (UID: "f1c26fbd-ffd1-438b-8c6a-36b667be335c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:41.831913 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.831895    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-token" pod "f1c26fbd-ffd1-438b-8c6a-36b667be335c" (UID: "f1c26fbd-ffd1-438b-8c6a-36b667be335c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:41.832267 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.832237    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c26fbd-ffd1-438b-8c6a-36b667be335c-kube-api-access-fsc9s" pod "f1c26fbd-ffd1-438b-8c6a-36b667be335c" (UID: "f1c26fbd-ffd1-438b-8c6a-36b667be335c"). InnerVolumeSpecName "kube-api-access-fsc9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:41.833245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.833210    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-podinfo" pod "f1c26fbd-ffd1-438b-8c6a-36b667be335c" (UID: "f1c26fbd-ffd1-438b-8c6a-36b667be335c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:25:41.863617 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.863590    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"]
Oct 30 22:25:41.868975 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.868951    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-476e593d-gw-openshift-default-789fdfb6d5-k5k6v"]
Oct 30 22:25:41.932892 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.932865    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:41.932892 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.932891    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:41.933038 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.932900    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1c26fbd-ffd1-438b-8c6a-36b667be335c-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:41.933038 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:41.932910    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsc9s\" (UniqueName: \"kubernetes.io/projected/f1c26fbd-ffd1-438b-8c6a-36b667be335c-kube-api-access-fsc9s\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:25:42.023169 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:42.023120    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:42.162760 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:42.162728    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"]
Oct 30 22:25:42.168243 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:42.168011    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-913eddcb-openshift-default-546bdfbdc5-ck6mz"]
Oct 30 22:25:42.182935 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:42.182574    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:42.701039 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:42.701012    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:42.784049 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:42.784014    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e967d2cf-50fa-4e9e-90b9-d49da1b81da1" path="/var/lib/kubelet/pods/e967d2cf-50fa-4e9e-90b9-d49da1b81da1/volumes"
Oct 30 22:25:42.784542 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:42.784526    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1c26fbd-ffd1-438b-8c6a-36b667be335c" path="/var/lib/kubelet/pods/f1c26fbd-ffd1-438b-8c6a-36b667be335c/volumes"
Oct 30 22:25:42.810610 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:42.810576    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx" event={"ID":"cc0df870-2a4a-404b-9678-bdc054dce3e3","Type":"ContainerStarted","Data":"dc70155ffafe9e975a8ca507d202284df36f0ae94460753a5267febbe8c22293"}
Oct 30 22:25:42.810781 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:42.810625    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx" event={"ID":"cc0df870-2a4a-404b-9678-bdc054dce3e3","Type":"ContainerStarted","Data":"a0716887e39b93d26bfbbca2c5dd96fdf78023de42306b53213e77d3f98e983a"}
Oct 30 22:25:42.810781 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:42.810761    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:42.811600 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:42.811580    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:25:42.853689 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:42.853641    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx" podStartSLOduration=2.853628017 podStartE2EDuration="2.853628017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:42.849347844 +0000 UTC m=+1128.205714158" watchObservedRunningTime="2026-10-30 22:25:42.853628017 +0000 UTC m=+1128.209994330"
Oct 30 22:25:43.382357 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:43.382321    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:43.547544 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:43.547505    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:43.818018 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:43.817987    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx"
Oct 30 22:25:44.001576 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:43.987937    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:45.038206 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.038171    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:45.116568 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.116538    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc"]
Oct 30 22:25:45.117258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.117245    4889 cpu_manager.go:414] "containerMap: removing container" podUID="e967d2cf-50fa-4e9e-90b9-d49da1b81da1" containerName="istio-proxy"
Oct 30 22:25:45.117310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.117259    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e967d2cf-50fa-4e9e-90b9-d49da1b81da1" containerName="istio-proxy"
Oct 30 22:25:45.117310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.117275    4889 cpu_manager.go:414] "containerMap: removing container" podUID="f1c26fbd-ffd1-438b-8c6a-36b667be335c" containerName="istio-proxy"
Oct 30 22:25:45.117310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.117280    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c26fbd-ffd1-438b-8c6a-36b667be335c" containerName="istio-proxy"
Oct 30 22:25:45.117408 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.117396    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="e967d2cf-50fa-4e9e-90b9-d49da1b81da1" containerName="istio-proxy"
Oct 30 22:25:45.117408 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.117408    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c26fbd-ffd1-438b-8c6a-36b667be335c" containerName="istio-proxy"
Oct 30 22:25:45.122442 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.122425    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc"
Oct 30 22:25:45.138548 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.138523    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc"]
Oct 30 22:25:45.141279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.141253    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-5983ea\""
Oct 30 22:25:45.146686 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.146660    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-5983ea-dockercfg-t9wnn\""
Oct 30 22:25:45.167688 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.167665    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k"]
Oct 30 22:25:45.172500 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.172483    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k"
Oct 30 22:25:45.192042 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.192016    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k"]
Oct 30 22:25:45.216421 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.216387    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:45.319302 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.306589    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de71837-99a5-4e01-ae12-385e722759eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k\" (UID: \"2de71837-99a5-4e01-ae12-385e722759eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k"
Oct 30 22:25:45.319302 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.306725    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/368f6471-b984-4935-92e6-047c8f81053f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc\" (UID: \"368f6471-b984-4935-92e6-047c8f81053f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc"
Oct 30 22:25:45.459257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.428794    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h798f\" (UniqueName: \"kubernetes.io/projected/368f6471-b984-4935-92e6-047c8f81053f-kube-api-access-h798f\") pod \"payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc\" (UID: \"368f6471-b984-4935-92e6-047c8f81053f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc"
Oct 30 22:25:45.459257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.428853    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chrhb\" (UniqueName: \"kubernetes.io/projected/2de71837-99a5-4e01-ae12-385e722759eb-kube-api-access-chrhb\") pod \"payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k\" (UID: \"2de71837-99a5-4e01-ae12-385e722759eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k"
Oct 30 22:25:45.554247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.533373    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de71837-99a5-4e01-ae12-385e722759eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k\" (UID: \"2de71837-99a5-4e01-ae12-385e722759eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k"
Oct 30 22:25:45.554247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.533523    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/368f6471-b984-4935-92e6-047c8f81053f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc\" (UID: \"368f6471-b984-4935-92e6-047c8f81053f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc"
Oct 30 22:25:45.554247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.533730    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h798f\" (UniqueName: \"kubernetes.io/projected/368f6471-b984-4935-92e6-047c8f81053f-kube-api-access-h798f\") pod \"payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc\" (UID: \"368f6471-b984-4935-92e6-047c8f81053f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc"
Oct 30 22:25:45.554247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.533767    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chrhb\" (UniqueName: \"kubernetes.io/projected/2de71837-99a5-4e01-ae12-385e722759eb-kube-api-access-chrhb\") pod \"payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k\" (UID: \"2de71837-99a5-4e01-ae12-385e722759eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k"
Oct 30 22:25:45.576295 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.556751    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de71837-99a5-4e01-ae12-385e722759eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k\" (UID: \"2de71837-99a5-4e01-ae12-385e722759eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k"
Oct 30 22:25:45.580380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.568145    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/368f6471-b984-4935-92e6-047c8f81053f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc\" (UID: \"368f6471-b984-4935-92e6-047c8f81053f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc"
Oct 30 22:25:45.605248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.603210    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h798f\" (UniqueName: \"kubernetes.io/projected/368f6471-b984-4935-92e6-047c8f81053f-kube-api-access-h798f\") pod \"payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc\" (UID: \"368f6471-b984-4935-92e6-047c8f81053f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc"
Oct 30 22:25:45.605248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.603745    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chrhb\" (UniqueName: \"kubernetes.io/projected/2de71837-99a5-4e01-ae12-385e722759eb-kube-api-access-chrhb\") pod \"payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k\" (UID: \"2de71837-99a5-4e01-ae12-385e722759eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k"
Oct 30 22:25:45.737242 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.737202    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc"
Oct 30 22:25:45.792837 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.792800    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k"
Oct 30 22:25:45.999395 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:45.970971    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:46.106196 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:46.106161    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc"]
Oct 30 22:25:46.106715 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:25:46.106689    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod368f6471_b984_4935_92e6_047c8f81053f.slice/crio-04d25166b2a14f36dac24eab3dde67574110b19ec02c047c1eb475e37b1606b3 WatchSource:0}: Error finding container 04d25166b2a14f36dac24eab3dde67574110b19ec02c047c1eb475e37b1606b3: Status 404 returned error can't find the container with id 04d25166b2a14f36dac24eab3dde67574110b19ec02c047c1eb475e37b1606b3
Oct 30 22:25:46.315605 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:25:46.315566    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de71837_99a5_4e01_ae12_385e722759eb.slice/crio-8233408b8fb3d926fc9c7bafe57ccd2da99600594233b519fac3d0746793583b WatchSource:0}: Error finding container 8233408b8fb3d926fc9c7bafe57ccd2da99600594233b519fac3d0746793583b: Status 404 returned error can't find the container with id 8233408b8fb3d926fc9c7bafe57ccd2da99600594233b519fac3d0746793583b
Oct 30 22:25:46.317619 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:46.317594    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k"]
Oct 30 22:25:46.592865 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:46.592830    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:46.846523 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:46.846483    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k" event={"ID":"2de71837-99a5-4e01-ae12-385e722759eb","Type":"ContainerStarted","Data":"f9de8cfaec16dc8ce77ddc7fd433efc4209f7a53eed48341dd0befa46505c4b5"}
Oct 30 22:25:46.846694 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:46.846555    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k"
Oct 30 22:25:46.846694 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:46.846580    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k" event={"ID":"2de71837-99a5-4e01-ae12-385e722759eb","Type":"ContainerStarted","Data":"8233408b8fb3d926fc9c7bafe57ccd2da99600594233b519fac3d0746793583b"}
Oct 30 22:25:46.848203 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:46.848178    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc" event={"ID":"368f6471-b984-4935-92e6-047c8f81053f","Type":"ContainerStarted","Data":"bb25d1a297429fc084984b461e011fcaa9b45b2995679e8d2e475ec02fe88956"}
Oct 30 22:25:46.848353 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:46.848219    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc" event={"ID":"368f6471-b984-4935-92e6-047c8f81053f","Type":"ContainerStarted","Data":"04d25166b2a14f36dac24eab3dde67574110b19ec02c047c1eb475e37b1606b3"}
Oct 30 22:25:46.848353 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:46.848275    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc"
Oct 30 22:25:46.886233 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:46.886176    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k" podStartSLOduration=1.88615746 podStartE2EDuration="1.88615746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:46.885057652 +0000 UTC m=+1132.241423976" watchObservedRunningTime="2026-10-30 22:25:46.88615746 +0000 UTC m=+1132.242523818"
Oct 30 22:25:46.925378 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:46.925333    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc" podStartSLOduration=1.9253187189999998 podStartE2EDuration="1.925318719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:46.923570275 +0000 UTC m=+1132.279936601" watchObservedRunningTime="2026-10-30 22:25:46.925318719 +0000 UTC m=+1132.281685033"
Oct 30 22:25:47.758251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:47.756566    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:49.389798 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:49.389761    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:50.791481 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:50.791450    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7"
Oct 30 22:25:50.791988 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:50.791532    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt"
Oct 30 22:25:51.144614 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:51.139704    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:52.794368 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:52.794336    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:52.991335 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:52.991299    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:54.682728 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:54.682690    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:54.827857 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:54.827823    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx"
Oct 30 22:25:56.528717 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:56.528678    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr"]
Oct 30 22:25:56.536265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:56.536197    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr"
Oct 30 22:25:56.551908 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:56.551874    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr"]
Oct 30 22:25:56.572568 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:56.572538    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-80ecb3-serving-cert\""
Oct 30 22:25:56.662410 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:56.662369    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7s9v\" (UniqueName: \"kubernetes.io/projected/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-kube-api-access-n7s9v\") pod \"maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr\" (UID: \"a3ff065c-81fb-4272-b8e1-0b8213ecbd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr"
Oct 30 22:25:56.662615 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:56.662433    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr\" (UID: \"a3ff065c-81fb-4272-b8e1-0b8213ecbd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr"
Oct 30 22:25:56.662615 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:56.662541    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr\" (UID: \"a3ff065c-81fb-4272-b8e1-0b8213ecbd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr"
Oct 30 22:25:56.662615 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:56.662585    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr\" (UID: \"a3ff065c-81fb-4272-b8e1-0b8213ecbd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr"
Oct 30 22:25:56.675304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:56.675275    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:56.764579 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:56.764537    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7s9v\" (UniqueName: \"kubernetes.io/projected/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-kube-api-access-n7s9v\") pod \"maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr\" (UID: \"a3ff065c-81fb-4272-b8e1-0b8213ecbd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr"
Oct 30 22:25:56.764860 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:56.764827    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr\" (UID: \"a3ff065c-81fb-4272-b8e1-0b8213ecbd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr"
Oct 30 22:25:56.765014 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:56.764874    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr\" (UID: \"a3ff065c-81fb-4272-b8e1-0b8213ecbd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr"
Oct 30 22:25:56.765014 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:56.764907    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr\" (UID: \"a3ff065c-81fb-4272-b8e1-0b8213ecbd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr"
Oct 30 22:25:56.765770 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:56.765709    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr\" (UID: \"a3ff065c-81fb-4272-b8e1-0b8213ecbd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr"
Oct 30 22:25:56.788303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:56.780760    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr\" (UID: \"a3ff065c-81fb-4272-b8e1-0b8213ecbd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr"
Oct 30 22:25:56.805145 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:56.805032    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7s9v\" (UniqueName: \"kubernetes.io/projected/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-kube-api-access-n7s9v\") pod \"maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr\" (UID: \"a3ff065c-81fb-4272-b8e1-0b8213ecbd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr"
Oct 30 22:25:56.834257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:56.828237    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr\" (UID: \"a3ff065c-81fb-4272-b8e1-0b8213ecbd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr"
Oct 30 22:25:56.870251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:56.860668    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr"
Oct 30 22:25:56.954555 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:56.954518    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:57.517248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:57.513883    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr"]
Oct 30 22:25:57.967248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:57.940436    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k"
Oct 30 22:25:57.972464 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:57.972433    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc"
Oct 30 22:25:58.273676 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:58.273567    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr" event={"ID":"a3ff065c-81fb-4272-b8e1-0b8213ecbd19","Type":"ContainerStarted","Data":"b4b7af42ffa74272055441b8f455e5e3a4b0f78fb870686e1dab840ba55d9d20"}
Oct 30 22:25:59.093409 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:59.093368    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:59.336655 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:59.336603    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr" event={"ID":"a3ff065c-81fb-4272-b8e1-0b8213ecbd19","Type":"ContainerStarted","Data":"74574b111d513140f217dba86f11e46e06a638aeccdfd084bf0ae0fd9c3d872d"}
Oct 30 22:25:59.337607 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:59.337577    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr"
Oct 30 22:25:59.389361 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:25:59.385615    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr" podStartSLOduration=2.623106813 podStartE2EDuration="3.385594366s" totalImagesPullingTime="762.487553ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:59.38340262 +0000 UTC m=+1144.739768934" watchObservedRunningTime="2026-10-30 22:25:59.385594366 +0000 UTC m=+1144.741960683"
Oct 30 22:26:00.936887 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:00.936827    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:02.578514 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:02.578471    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:02.813431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:02.813394    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:02.978317 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:02.978209    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:04.545542 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:04.545495    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:06.350611 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:06.350566    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:06.393621 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:06.393592    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr"
Oct 30 22:26:08.046254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:08.038671    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:09.705869 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:09.705814    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:10.301757 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:26:10.301722    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:26:11.547884 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:11.544964    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:13.251248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:13.245751    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:15.284249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:15.283511    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:17.040252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:17.038639    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:18.946641 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:18.946604    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:19.596695 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:19.596657    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:20.055827 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:20.055793    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:20.504915 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:20.504880    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:20.670531 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:20.670480    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:20.874746 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:20.874660    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:21.063266 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:21.059163    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:22.926588 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:22.926549    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:23.096716 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:23.096680    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:23.946958 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:23.946913    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:24.116153 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:24.116118    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:25.226337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:25.226218    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:27.127468 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:27.127434    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:27.240900 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:27.240868    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9"]
Oct 30 22:26:27.246836 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:27.246815    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9"
Oct 30 22:26:27.263113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:27.263082    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9"]
Oct 30 22:26:27.269499 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:27.269468    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-3cd77f\""
Oct 30 22:26:27.269673 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:27.269562    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-3cd77f-dockercfg-vk2ds\""
Oct 30 22:26:27.290033 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:27.290004    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:27.291097 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:27.291074    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rh9l\" (UniqueName: \"kubernetes.io/projected/b37648ec-d503-43af-aef6-ec9da60a0902-kube-api-access-8rh9l\") pod \"payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9\" (UID: \"b37648ec-d503-43af-aef6-ec9da60a0902\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9"
Oct 30 22:26:27.291252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:27.291165    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37648ec-d503-43af-aef6-ec9da60a0902-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9\" (UID: \"b37648ec-d503-43af-aef6-ec9da60a0902\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9"
Oct 30 22:26:27.348393 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:27.348361    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd"]
Oct 30 22:26:27.352776 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:27.352736    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd"
Oct 30 22:26:27.386065 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:27.385946    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd"]
Oct 30 22:26:27.401987 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:27.401929    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37648ec-d503-43af-aef6-ec9da60a0902-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9\" (UID: \"b37648ec-d503-43af-aef6-ec9da60a0902\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9"
Oct 30 22:26:27.402361 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:27.402330    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/341cd65b-6e79-41b5-996c-1b866e57780c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd\" (UID: \"341cd65b-6e79-41b5-996c-1b866e57780c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd"
Oct 30 22:26:27.402888 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:27.402863    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6kcb\" (UniqueName: \"kubernetes.io/projected/341cd65b-6e79-41b5-996c-1b866e57780c-kube-api-access-b6kcb\") pod \"payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd\" (UID: \"341cd65b-6e79-41b5-996c-1b866e57780c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd"
Oct 30 22:26:27.403145 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:27.403129    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rh9l\" (UniqueName: \"kubernetes.io/projected/b37648ec-d503-43af-aef6-ec9da60a0902-kube-api-access-8rh9l\") pod \"payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9\" (UID: \"b37648ec-d503-43af-aef6-ec9da60a0902\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9"
Oct 30 22:26:27.404685 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:27.404601    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37648ec-d503-43af-aef6-ec9da60a0902-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9\" (UID: \"b37648ec-d503-43af-aef6-ec9da60a0902\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9"
Oct 30 22:26:27.577249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:27.575035    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/341cd65b-6e79-41b5-996c-1b866e57780c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd\" (UID: \"341cd65b-6e79-41b5-996c-1b866e57780c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd"
Oct 30 22:26:27.670405 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:27.657248    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/341cd65b-6e79-41b5-996c-1b866e57780c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd\" (UID: \"341cd65b-6e79-41b5-996c-1b866e57780c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd"
Oct 30 22:26:27.670405 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:27.657376    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6kcb\" (UniqueName: \"kubernetes.io/projected/341cd65b-6e79-41b5-996c-1b866e57780c-kube-api-access-b6kcb\") pod \"payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd\" (UID: \"341cd65b-6e79-41b5-996c-1b866e57780c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd"
Oct 30 22:26:27.779777 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:27.751443    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rh9l\" (UniqueName: \"kubernetes.io/projected/b37648ec-d503-43af-aef6-ec9da60a0902-kube-api-access-8rh9l\") pod \"payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9\" (UID: \"b37648ec-d503-43af-aef6-ec9da60a0902\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9"
Oct 30 22:26:27.779777 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:27.766587    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6kcb\" (UniqueName: \"kubernetes.io/projected/341cd65b-6e79-41b5-996c-1b866e57780c-kube-api-access-b6kcb\") pod \"payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd\" (UID: \"341cd65b-6e79-41b5-996c-1b866e57780c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd"
Oct 30 22:26:27.874257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:27.865776    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9"
Oct 30 22:26:27.980380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:27.980326    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd"
Oct 30 22:26:28.278344 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:28.278061    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9"]
Oct 30 22:26:28.608714 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:28.608683    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd"]
Oct 30 22:26:28.908292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:28.908254    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd" event={"ID":"341cd65b-6e79-41b5-996c-1b866e57780c","Type":"ContainerStarted","Data":"a800fe6a8039be3f172b17739d13e27ef824ebb2b005f5f0ad2eef0499d2815e"}
Oct 30 22:26:28.922205 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:28.922164    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9" event={"ID":"b37648ec-d503-43af-aef6-ec9da60a0902","Type":"ContainerStarted","Data":"0b0e660e33109897d30c5f9f6d29a88f0f7a8b700186279151228cccb1533f3d"}
Oct 30 22:26:28.922467 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:28.922445    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9" event={"ID":"b37648ec-d503-43af-aef6-ec9da60a0902","Type":"ContainerStarted","Data":"976e42b879a10b11250293188940f7c38198305512f92d27b32c2a7756fc0371"}
Oct 30 22:26:28.923784 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:28.923762    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9"
Oct 30 22:26:29.005427 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:29.002491    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:29.005427 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:29.004932    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9" podStartSLOduration=2.004915232 podStartE2EDuration="2.004915232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:28.998722906 +0000 UTC m=+1174.355089221" watchObservedRunningTime="2026-10-30 22:26:29.004915232 +0000 UTC m=+1174.361281547"
Oct 30 22:26:29.267252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:29.264711    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:29.553468 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:29.550847    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:29.880302 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:29.861308    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"]
Oct 30 22:26:30.135317 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.121306    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:30.432004 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.431911    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:30.502249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.475477    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"]
Oct 30 22:26:30.502577 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.502552    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd"
Oct 30 22:26:30.511280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.502716    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd" event={"ID":"341cd65b-6e79-41b5-996c-1b866e57780c","Type":"ContainerStarted","Data":"19a2d32ad49e78a0353850ac47e9854be819d22c5cf249b26249a4cdef9eeeaa"}
Oct 30 22:26:30.550513 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.550483    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:30.572415 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.572386    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-6af86e-openshift-default-dockercfg-lgnvg\""
Oct 30 22:26:30.602787 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.602730    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd" podStartSLOduration=3.60270961 podStartE2EDuration="3.60270961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:30.486296213 +0000 UTC m=+1175.842662529" watchObservedRunningTime="2026-10-30 22:26:30.60270961 +0000 UTC m=+1175.959075931"
Oct 30 22:26:30.639821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.620515    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istiod-ca-cert\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:30.641095 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.640973    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-workload-certs\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:30.641417 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.641394    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-token\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:30.641636 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.641621    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-podinfo\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:30.641783 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.641770    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-workload-socket\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:30.641878 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.641866    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-data\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:30.642006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.641994    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-envoy\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:30.642132 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.642118    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-credential-socket\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:30.642249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.642214    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5dtk\" (UniqueName: \"kubernetes.io/projected/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-kube-api-access-x5dtk\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:30.744506 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.744464    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-workload-certs\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:30.745263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.745239    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-token\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:30.760260 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.745519    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-podinfo\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:30.808249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.745139    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-workload-certs\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:30.808249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.800057    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-workload-socket\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:30.808249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.800096    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-data\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:30.808249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.800179    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-envoy\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:30.808249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.800268    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-credential-socket\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:30.808249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.800315    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5dtk\" (UniqueName: \"kubernetes.io/projected/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-kube-api-access-x5dtk\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:30.808249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.800673    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istiod-ca-cert\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:30.808249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.801530    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istiod-ca-cert\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:30.808249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.801809    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-workload-socket\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:30.808249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.802040    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-data\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:30.808249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.802670    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-credential-socket\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:30.873290 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.865201    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-envoy\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:30.893258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.876721    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-token\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:30.902253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.894150    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-podinfo\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:30.921599 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:30.921566    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5dtk\" (UniqueName: \"kubernetes.io/projected/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-kube-api-access-x5dtk\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:31.011916 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:31.011594    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:31.170327 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:31.159497    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:26:31.239252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:31.219666    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:26:31.262247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:31.261324    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:26:31.288381 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:31.288191    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:26:31.299820 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:31.299784    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:26:31.902871 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:31.902837    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"]
Oct 30 22:26:31.944295 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:31.944195    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:26:31.944677 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:31.944636    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:26:31.953247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:31.944860    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:26:32.066250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:31.987024    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:32.638253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:32.623370    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj" event={"ID":"d7ce41e7-a1cb-4f49-a247-da6c57d34d53","Type":"ContainerStarted","Data":"f52b786cc3ba6ab73873173d3b67ec58ee39c663f597f780bb9e70b04ae8225a"}
Oct 30 22:26:32.639877 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:32.639843    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj" event={"ID":"d7ce41e7-a1cb-4f49-a247-da6c57d34d53","Type":"ContainerStarted","Data":"32dadc9915f448544eeedcce8766c3806dedfbc3acfa092e9d25f968273fac51"}
Oct 30 22:26:32.704311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:32.704153    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj" podStartSLOduration=3.704136695 podStartE2EDuration="3.704136695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:32.689155845 +0000 UTC m=+1178.045522159" watchObservedRunningTime="2026-10-30 22:26:32.704136695 +0000 UTC m=+1178.060503011"
Oct 30 22:26:33.084265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:33.073012    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:26:33.084265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:33.073074    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj" podUID="d7ce41e7-a1cb-4f49-a247-da6c57d34d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 22:26:33.109051 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:33.109007    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:34.063249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:34.062884    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:34.263250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:34.262695    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:34.706968 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:34.706896    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:34.708207 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:34.707933    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:26:35.190989 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:35.190960    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:35.358012 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:35.357922    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:36.318248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:36.315779    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:36.860876 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:36.860672    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:37.015486 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:37.014540    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b"]
Oct 30 22:26:37.032474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:37.032287    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:37.352921 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:37.352885    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b"
Oct 30 22:26:37.370297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:37.370266    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-6af86e-dockercfg-6kzwt\""
Oct 30 22:26:37.383596 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:37.383566    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-6af86e\""
Oct 30 22:26:37.399743 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:37.398648    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b"]
Oct 30 22:26:37.443264 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:37.433149    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh8d2\" (UniqueName: \"kubernetes.io/projected/dafe75fd-77ad-49d3-aafb-35f28fcb2954-kube-api-access-bh8d2\") pod \"payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b\" (UID: \"dafe75fd-77ad-49d3-aafb-35f28fcb2954\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b"
Oct 30 22:26:37.443264 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:37.433443    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dafe75fd-77ad-49d3-aafb-35f28fcb2954-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b\" (UID: \"dafe75fd-77ad-49d3-aafb-35f28fcb2954\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b"
Oct 30 22:26:37.535667 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:37.535633    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dafe75fd-77ad-49d3-aafb-35f28fcb2954-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b\" (UID: \"dafe75fd-77ad-49d3-aafb-35f28fcb2954\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b"
Oct 30 22:26:37.536052 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:37.536037    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh8d2\" (UniqueName: \"kubernetes.io/projected/dafe75fd-77ad-49d3-aafb-35f28fcb2954-kube-api-access-bh8d2\") pod \"payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b\" (UID: \"dafe75fd-77ad-49d3-aafb-35f28fcb2954\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b"
Oct 30 22:26:37.536771 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:37.536746    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dafe75fd-77ad-49d3-aafb-35f28fcb2954-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b\" (UID: \"dafe75fd-77ad-49d3-aafb-35f28fcb2954\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b"
Oct 30 22:26:37.561325 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:37.561286    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh8d2\" (UniqueName: \"kubernetes.io/projected/dafe75fd-77ad-49d3-aafb-35f28fcb2954-kube-api-access-bh8d2\") pod \"payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b\" (UID: \"dafe75fd-77ad-49d3-aafb-35f28fcb2954\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b"
Oct 30 22:26:37.758079 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:37.758041    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b"
Oct 30 22:26:38.092603 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:38.092530    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b"]
Oct 30 22:26:38.102004 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:26:38.101501    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddafe75fd_77ad_49d3_aafb_35f28fcb2954.slice/crio-ffef39e2aab3f7ddcf1140535d7f36ac30e6c5e5c79e9f1aad96976dd59f9da2 WatchSource:0}: Error finding container ffef39e2aab3f7ddcf1140535d7f36ac30e6c5e5c79e9f1aad96976dd59f9da2: Status 404 returned error can't find the container with id ffef39e2aab3f7ddcf1140535d7f36ac30e6c5e5c79e9f1aad96976dd59f9da2
Oct 30 22:26:38.385660 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:38.382048    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:38.748405 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:38.748363    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b" event={"ID":"dafe75fd-77ad-49d3-aafb-35f28fcb2954","Type":"ContainerStarted","Data":"21096e1f1e587ecd0afcb91eaefabb23562d761057706773c44598c9a8484c57"}
Oct 30 22:26:38.748591 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:38.748437    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b" event={"ID":"dafe75fd-77ad-49d3-aafb-35f28fcb2954","Type":"ContainerStarted","Data":"ffef39e2aab3f7ddcf1140535d7f36ac30e6c5e5c79e9f1aad96976dd59f9da2"}
Oct 30 22:26:38.748591 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:38.748474    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b"
Oct 30 22:26:38.756396 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:38.756369    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:26:38.756526 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:38.756420    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n" podUID="9833755d-9e30-41f6-806f-688a9cb72349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:26:38.790988 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:38.790908    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b" podStartSLOduration=2.7908904039999998 podStartE2EDuration="2.790890404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:38.790729031 +0000 UTC m=+1184.147095351" watchObservedRunningTime="2026-10-30 22:26:38.790890404 +0000 UTC m=+1184.147256718"
Oct 30 22:26:40.159656 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:40.156025    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:40.469917 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:26:40.303119    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:26:41.605628 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:41.605604    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9"
Oct 30 22:26:41.776331 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:41.776304    4889 generic.go:356] "Generic (PLEG): container finished" podID="c02298a7-464d-4cc4-8bef-3be1dfc6f3ab" containerID="acee3959f7a8ff1c1b5e6f4e56a3c97046c0955298ab9883b0a770de90546c16" exitCode=0
Oct 30 22:26:41.776470 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:41.776366    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg" event={"ID":"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab","Type":"ContainerDied","Data":"acee3959f7a8ff1c1b5e6f4e56a3c97046c0955298ab9883b0a770de90546c16"}
Oct 30 22:26:41.776720 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:41.776703    4889 scope.go:122] "RemoveContainer" containerID="acee3959f7a8ff1c1b5e6f4e56a3c97046c0955298ab9883b0a770de90546c16"
Oct 30 22:26:41.778300 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:41.778270    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:26:41.778383 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:41.778340    4889 generic.go:356] "Generic (PLEG): container finished" podID="9833755d-9e30-41f6-806f-688a9cb72349" containerID="da0e6ca092e95570b9a393d80f2b5144ef5056dd31dbe09c02b06910de65c043" exitCode=0
Oct 30 22:26:41.778383 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:41.778354    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:26:41.778448 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:41.778400    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:26:41.778448 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:41.778406    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n" event={"ID":"9833755d-9e30-41f6-806f-688a9cb72349","Type":"ContainerDied","Data":"da0e6ca092e95570b9a393d80f2b5144ef5056dd31dbe09c02b06910de65c043"}
Oct 30 22:26:41.778704 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:41.778691    4889 scope.go:122] "RemoveContainer" containerID="da0e6ca092e95570b9a393d80f2b5144ef5056dd31dbe09c02b06910de65c043"
Oct 30 22:26:41.780301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:41.780275    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:26:41.780367 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:41.780351    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:26:41.780416 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:41.780402    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:26:41.935834 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:41.935691    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:42.648486 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:42.648456    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd"
Oct 30 22:26:42.832390 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:42.832354    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:26:42.879243 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:42.878961    4889 generic.go:356] "Generic (PLEG): container finished" podID="62c75d3c-bba8-4a52-a0cf-f76a2b4b8495" containerID="8595aa86978856d85bb6582032e10dcd5561f2ba65c802c5800c469f310a76bd" exitCode=0
Oct 30 22:26:42.879243 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:42.879023    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn" event={"ID":"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495","Type":"ContainerDied","Data":"8595aa86978856d85bb6582032e10dcd5561f2ba65c802c5800c469f310a76bd"}
Oct 30 22:26:42.879550 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:42.879528    4889 scope.go:122] "RemoveContainer" containerID="8595aa86978856d85bb6582032e10dcd5561f2ba65c802c5800c469f310a76bd"
Oct 30 22:26:42.881954 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:42.881901    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:26:42.882610 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:42.882573    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:26:42.883370 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:42.883341    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:26:42.883650 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:42.883625    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg" event={"ID":"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab","Type":"ContainerStarted","Data":"be3cd57e007b64c0705a95159a6611725bdd672c86bd63fb3d1e9bc05d1cfc3b"}
Oct 30 22:26:42.886473 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:42.886389    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n" event={"ID":"9833755d-9e30-41f6-806f-688a9cb72349","Type":"ContainerStarted","Data":"e9ddae6b8ec8a39d5acccee392f68000c1d6d6481c28cadb5f98ac92eb91b5c8"}
Oct 30 22:26:43.166469 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:43.166376    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:26:43.166652 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:43.166622    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:26:43.166726 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:43.166693    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg" podUID="c02298a7-464d-4cc4-8bef-3be1dfc6f3ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 22:26:43.276389 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:43.276351    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:26:43.276594 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:43.276471    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:26:43.276594 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:43.276523    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n" podUID="9833755d-9e30-41f6-806f-688a9cb72349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 22:26:43.758857 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:43.758821    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:44.058634 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:44.058507    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn" event={"ID":"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495","Type":"ContainerStarted","Data":"73a127efa766b7a42a4d634d676dc9b91d384681d01ab479f1af4721ac308b13"}
Oct 30 22:26:44.167611 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:44.167573    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:26:44.167885 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:44.167853    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg" podUID="c02298a7-464d-4cc4-8bef-3be1dfc6f3ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 22:26:44.283404 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:44.278873    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:26:44.283404 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:44.278932    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n" podUID="9833755d-9e30-41f6-806f-688a9cb72349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 22:26:44.528560 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:44.528529    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:26:44.528973 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:44.528949    4889 patch_prober.go:29] interesting pod/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:26:44.529148 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:44.529124    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn" podUID="62c75d3c-bba8-4a52-a0cf-f76a2b4b8495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 22:26:45.166251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:45.165604    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:26:45.166251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:45.165683    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg" podUID="c02298a7-464d-4cc4-8bef-3be1dfc6f3ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 22:26:45.292430 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:45.279616    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:26:45.292430 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:45.279690    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n" podUID="9833755d-9e30-41f6-806f-688a9cb72349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 22:26:45.532664 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:45.532620    4889 patch_prober.go:29] interesting pod/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:26:45.532951 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:45.532922    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn" podUID="62c75d3c-bba8-4a52-a0cf-f76a2b4b8495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 22:26:45.803037 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:45.792666    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:46.050277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:46.050236    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:46.167290 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:46.167106    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:26:46.167966 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:46.167839    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg" podUID="c02298a7-464d-4cc4-8bef-3be1dfc6f3ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 22:26:46.251279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:46.238111    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:46.286410 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:46.286376    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:26:46.286686 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:46.286658    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n" podUID="9833755d-9e30-41f6-806f-688a9cb72349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 22:26:46.555824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:46.555786    4889 patch_prober.go:29] interesting pod/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:26:46.556744 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:46.556709    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn" podUID="62c75d3c-bba8-4a52-a0cf-f76a2b4b8495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 22:26:46.792259 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:46.790867    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:47.166038 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:47.165993    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:26:47.166242 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:47.166079    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg" podUID="c02298a7-464d-4cc4-8bef-3be1dfc6f3ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 22:26:47.295247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:47.292723    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:26:47.295247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:47.292799    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n" podUID="9833755d-9e30-41f6-806f-688a9cb72349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 22:26:47.527250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:47.527160    4889 patch_prober.go:29] interesting pod/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:26:47.527440 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:47.527258    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn" podUID="62c75d3c-bba8-4a52-a0cf-f76a2b4b8495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 22:26:48.169250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:48.169121    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:26:48.169250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:48.169195    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg" podUID="c02298a7-464d-4cc4-8bef-3be1dfc6f3ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 22:26:48.286253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:48.281553    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:26:48.286253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:48.281648    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n" podUID="9833755d-9e30-41f6-806f-688a9cb72349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 22:26:48.530738 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:48.530702    4889 patch_prober.go:29] interesting pod/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:26:48.531401 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:48.531349    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn" podUID="62c75d3c-bba8-4a52-a0cf-f76a2b4b8495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 22:26:48.608065 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:48.608028    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:48.718393 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:48.718367    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:26:49.165500 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:49.165463    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:26:49.165683 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:49.165530    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg" podUID="c02298a7-464d-4cc4-8bef-3be1dfc6f3ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 22:26:49.281037 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:49.280997    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:26:49.281519 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:49.281501    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:26:49.282351 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:49.282327    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 22:26:49.544631 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:49.544598    4889 patch_prober.go:29] interesting pod/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:26:49.545189 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:49.545158    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn" podUID="62c75d3c-bba8-4a52-a0cf-f76a2b4b8495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 22:26:49.549124 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:49.549099    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:49.737316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:49.734775    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:49.761023 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:49.760998    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b"
Oct 30 22:26:50.171199 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:50.171168    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:26:50.171878 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:50.171845    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:26:50.172757 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:50.172731    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 22:26:50.490300 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:50.487005    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:50.560654 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:50.560615    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:26:50.583737 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:50.583708    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:26:52.235385 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:52.235345    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:53.901159 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:53.901115    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:55.043530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:55.043505    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 22:26:55.047151 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:55.047127    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 22:26:55.690250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:55.685312    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:57.686570 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:57.686524    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:57.873835 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:57.873806    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:59.536905 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:26:59.536873    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:01.289248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:01.281535    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:03.045539 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:03.045498    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:04.844937 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:04.844859    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:05.998796 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:05.998757    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:06.469641 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:06.469543    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:06.689757 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:06.689724    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:08.238337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:08.238305    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:09.303158 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:09.303119    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:09.795249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:09.790914    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:09.971248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:09.969854    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:10.355945 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:27:10.355911    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:27:10.649838 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:10.649753    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:10.835957 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:10.835925    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:12.335184 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:12.335152    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:12.376728 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:12.376701    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb"]
Oct 30 22:27:12.381478 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:12.381429    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb"
Oct 30 22:27:12.399427 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:12.399400    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb"]
Oct 30 22:27:12.405115 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:12.405090    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d399fb07\""
Oct 30 22:27:12.405514 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:12.405464    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d399fb07-dockercfg-mrr66\""
Oct 30 22:27:12.471870 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:12.471835    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzpc4\" (UniqueName: \"kubernetes.io/projected/77442c89-ba79-4612-85eb-cab9b6cc702c-kube-api-access-vzpc4\") pod \"payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb\" (UID: \"77442c89-ba79-4612-85eb-cab9b6cc702c\") " pod="openshift-ingress/payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb"
Oct 30 22:27:12.476609 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:12.476578    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77442c89-ba79-4612-85eb-cab9b6cc702c-plugins-config-volume\") pod \"payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb\" (UID: \"77442c89-ba79-4612-85eb-cab9b6cc702c\") " pod="openshift-ingress/payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb"
Oct 30 22:27:12.528856 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:12.528819    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:12.634292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:12.634190    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzpc4\" (UniqueName: \"kubernetes.io/projected/77442c89-ba79-4612-85eb-cab9b6cc702c-kube-api-access-vzpc4\") pod \"payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb\" (UID: \"77442c89-ba79-4612-85eb-cab9b6cc702c\") " pod="openshift-ingress/payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb"
Oct 30 22:27:12.661249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:12.635217    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77442c89-ba79-4612-85eb-cab9b6cc702c-plugins-config-volume\") pod \"payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb\" (UID: \"77442c89-ba79-4612-85eb-cab9b6cc702c\") " pod="openshift-ingress/payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb"
Oct 30 22:27:12.662243 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:12.647603    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77442c89-ba79-4612-85eb-cab9b6cc702c-plugins-config-volume\") pod \"payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb\" (UID: \"77442c89-ba79-4612-85eb-cab9b6cc702c\") " pod="openshift-ingress/payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb"
Oct 30 22:27:12.836013 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:12.835978    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzpc4\" (UniqueName: \"kubernetes.io/projected/77442c89-ba79-4612-85eb-cab9b6cc702c-kube-api-access-vzpc4\") pod \"payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb\" (UID: \"77442c89-ba79-4612-85eb-cab9b6cc702c\") " pod="openshift-ingress/payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb"
Oct 30 22:27:13.000835 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:13.000805    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb"
Oct 30 22:27:13.201251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:13.200210    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:13.340304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:13.334499    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb"]
Oct 30 22:27:13.343406 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:27:13.343378    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77442c89_ba79_4612_85eb_cab9b6cc702c.slice/crio-7c86b1700f0fa37cc970e98b2040e71ad2c29d5e2ddbb470c6c45a19aee506d4 WatchSource:0}: Error finding container 7c86b1700f0fa37cc970e98b2040e71ad2c29d5e2ddbb470c6c45a19aee506d4: Status 404 returned error can't find the container with id 7c86b1700f0fa37cc970e98b2040e71ad2c29d5e2ddbb470c6c45a19aee506d4
Oct 30 22:27:13.657691 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:13.657613    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:13.824008 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:13.823973    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:13.879554 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:13.879518    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb" event={"ID":"77442c89-ba79-4612-85eb-cab9b6cc702c","Type":"ContainerStarted","Data":"f77b786bc6abbfe05720843479c226296fdcbef7601c9e691d2a29d4eaae67aa"}
Oct 30 22:27:13.879719 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:13.879582    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb" event={"ID":"77442c89-ba79-4612-85eb-cab9b6cc702c","Type":"ContainerStarted","Data":"7c86b1700f0fa37cc970e98b2040e71ad2c29d5e2ddbb470c6c45a19aee506d4"}
Oct 30 22:27:13.879719 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:13.879614    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb"
Oct 30 22:27:13.922650 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:13.922556    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb" podStartSLOduration=1.922540976 podStartE2EDuration="1.922540976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:13.92021917 +0000 UTC m=+1219.276585483" watchObservedRunningTime="2026-10-30 22:27:13.922540976 +0000 UTC m=+1219.278907289"
Oct 30 22:27:14.802939 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:14.802896    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:15.251300 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:15.249486    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:15.436931 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:15.436779    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:15.463525 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:15.463481    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s"]
Oct 30 22:27:15.468611 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:15.468589    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s"
Oct 30 22:27:15.489216 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:15.489189    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s"]
Oct 30 22:27:15.500338 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:15.500287    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-6f2eba-dockercfg-m8m59\""
Oct 30 22:27:15.674312 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:15.622962    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-6f2eba\""
Oct 30 22:27:15.704589 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:15.704516    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:15.731394 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:15.731355    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz9qm\" (UniqueName: \"kubernetes.io/projected/c54084a2-132f-437f-a13d-42f118c87251-kube-api-access-jz9qm\") pod \"payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s\" (UID: \"c54084a2-132f-437f-a13d-42f118c87251\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s"
Oct 30 22:27:15.749428 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:15.731916    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c54084a2-132f-437f-a13d-42f118c87251-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s\" (UID: \"c54084a2-132f-437f-a13d-42f118c87251\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s"
Oct 30 22:27:15.873307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:15.851342    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c54084a2-132f-437f-a13d-42f118c87251-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s\" (UID: \"c54084a2-132f-437f-a13d-42f118c87251\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s"
Oct 30 22:27:15.941851 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:15.941767    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz9qm\" (UniqueName: \"kubernetes.io/projected/c54084a2-132f-437f-a13d-42f118c87251-kube-api-access-jz9qm\") pod \"payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s\" (UID: \"c54084a2-132f-437f-a13d-42f118c87251\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s"
Oct 30 22:27:15.944843 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:15.874905    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c54084a2-132f-437f-a13d-42f118c87251-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s\" (UID: \"c54084a2-132f-437f-a13d-42f118c87251\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s"
Oct 30 22:27:15.990420 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:15.990389    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz9qm\" (UniqueName: \"kubernetes.io/projected/c54084a2-132f-437f-a13d-42f118c87251-kube-api-access-jz9qm\") pod \"payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s\" (UID: \"c54084a2-132f-437f-a13d-42f118c87251\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s"
Oct 30 22:27:16.092855 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:16.092821    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:16.097772 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:16.097746    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s"
Oct 30 22:27:16.288971 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:16.288937    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:16.409672 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:27:16.409640    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc54084a2_132f_437f_a13d_42f118c87251.slice/crio-60574b61ad7fc0e4d483931228b2e8193dd23fbfdf462e8247ab348968e0ce64 WatchSource:0}: Error finding container 60574b61ad7fc0e4d483931228b2e8193dd23fbfdf462e8247ab348968e0ce64: Status 404 returned error can't find the container with id 60574b61ad7fc0e4d483931228b2e8193dd23fbfdf462e8247ab348968e0ce64
Oct 30 22:27:16.410423 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:16.410401    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s"]
Oct 30 22:27:16.951753 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:16.951715    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s" event={"ID":"c54084a2-132f-437f-a13d-42f118c87251","Type":"ContainerStarted","Data":"f5bf34517fb525f2b88f680a02c7940bcc5a7050060bd7654a8e39a326904b04"}
Oct 30 22:27:16.952143 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:16.951779    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s" event={"ID":"c54084a2-132f-437f-a13d-42f118c87251","Type":"ContainerStarted","Data":"60574b61ad7fc0e4d483931228b2e8193dd23fbfdf462e8247ab348968e0ce64"}
Oct 30 22:27:16.952143 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:16.951807    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s"
Oct 30 22:27:17.003617 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:17.003563    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s" podStartSLOduration=2.003547378 podStartE2EDuration="2.003547378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:16.99951482 +0000 UTC m=+1222.355881126" watchObservedRunningTime="2026-10-30 22:27:17.003547378 +0000 UTC m=+1222.359913705"
Oct 30 22:27:17.259520 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:17.259471    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:17.438573 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:17.438539    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:17.738578 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:17.738494    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:19.664550 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:19.662896    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:22.404024 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:22.403989    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:23.168567 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:23.168531    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:24.899247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:24.897125    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb"
Oct 30 22:27:25.199307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:25.196976    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:26.065882 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:26.065844    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:27.047357 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:27.045741    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:27.127249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:27.116283    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7266-24ec6b"]
Oct 30 22:27:27.182032 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:27.181997    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7266-24ec6b"
Oct 30 22:27:27.187191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:27.187148    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7266-24ec6b"]
Oct 30 22:27:27.199653 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:27.198887    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qxqt\" (UniqueName: \"kubernetes.io/projected/0f4160c3-69cd-4944-92d0-0e5690de1093-kube-api-access-2qxqt\") pod \"test-curl-7266-24ec6b\" (UID: \"0f4160c3-69cd-4944-92d0-0e5690de1093\") " pod="opendatahub/test-curl-7266-24ec6b"
Oct 30 22:27:27.200444 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:27.200414    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xbw52\""
Oct 30 22:27:27.301122 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:27.301040    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qxqt\" (UniqueName: \"kubernetes.io/projected/0f4160c3-69cd-4944-92d0-0e5690de1093-kube-api-access-2qxqt\") pod \"test-curl-7266-24ec6b\" (UID: \"0f4160c3-69cd-4944-92d0-0e5690de1093\") " pod="opendatahub/test-curl-7266-24ec6b"
Oct 30 22:27:27.320748 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:27.320722    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qxqt\" (UniqueName: \"kubernetes.io/projected/0f4160c3-69cd-4944-92d0-0e5690de1093-kube-api-access-2qxqt\") pod \"test-curl-7266-24ec6b\" (UID: \"0f4160c3-69cd-4944-92d0-0e5690de1093\") " pod="opendatahub/test-curl-7266-24ec6b"
Oct 30 22:27:27.506214 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:27.506180    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7266-24ec6b"
Oct 30 22:27:27.812200 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:27:27.812168    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f4160c3_69cd_4944_92d0_0e5690de1093.slice/crio-04c712c6d1c224ddecf7f7ed2637ee8c050fe0902ade4e7ba92068608a5d5335 WatchSource:0}: Error finding container 04c712c6d1c224ddecf7f7ed2637ee8c050fe0902ade4e7ba92068608a5d5335: Status 404 returned error can't find the container with id 04c712c6d1c224ddecf7f7ed2637ee8c050fe0902ade4e7ba92068608a5d5335
Oct 30 22:27:27.814559 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:27.814454    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7266-24ec6b"]
Oct 30 22:27:27.964748 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:27.964720    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s"
Oct 30 22:27:28.105639 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:28.105555    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7266-24ec6b" event={"ID":"0f4160c3-69cd-4944-92d0-0e5690de1093","Type":"ContainerStarted","Data":"04c712c6d1c224ddecf7f7ed2637ee8c050fe0902ade4e7ba92068608a5d5335"}
Oct 30 22:27:28.725980 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:28.725078    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:29.565241 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:29.565196    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:30.985956 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:30.985921    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:33.352585 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:33.349709    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:33.408331 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:33.402858    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:33.452309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:33.436029    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:33.632420 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:33.632315    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded"
Oct 30 22:27:33.686249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:33.669066    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:27:33.702751 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:33.702694    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:33.823250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:33.806317    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:33.903298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:33.901542    4889 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded"
Oct 30 22:27:34.168822 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:34.168721    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:34.254366 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:34.254318    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:34.254750 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:34.254620    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:34.325933 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:34.325353    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:34.462306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:34.449485    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:34.462306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:34.449685    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:27:34.484338 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:34.484301    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:34.485390 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:34.485366    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:27:34.589910 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:34.589882    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:27:34.673643 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:34.673614    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:27:35.557248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:35.546970    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:36.802743 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:36.802711    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7266-24ec6b" event={"ID":"0f4160c3-69cd-4944-92d0-0e5690de1093","Type":"ContainerStarted","Data":"b62b9d2b169586489c2f1af7c3be37a63b3282914e1b03bdb7280a4266d7d774"}
Oct 30 22:27:36.857525 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:36.857475    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7266-24ec6b" podStartSLOduration=2.178878821 podStartE2EDuration="9.857460332s" totalImagesPullingTime="7.678581511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:36.856415032 +0000 UTC m=+1242.212781346" watchObservedRunningTime="2026-10-30 22:27:36.857460332 +0000 UTC m=+1242.213826645"
Oct 30 22:27:36.927119 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:36.927085    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:37.483922 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:37.483836    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:37.675296 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:37.664251    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:37.890888 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:37.890858    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7266-24ec6b"]
Oct 30 22:27:37.898493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:37.898457    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7266-24ec6b"]
Oct 30 22:27:37.998767 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:37.998735    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:38.039602 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:38.039565    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7266-c9f195"]
Oct 30 22:27:38.040424 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:38.040406    4889 cpu_manager.go:414] "containerMap: removing container" podUID="0f4160c3-69cd-4944-92d0-0e5690de1093" containerName="test-curl-7266-24ec6b"
Oct 30 22:27:38.040424 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:38.040425    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f4160c3-69cd-4944-92d0-0e5690de1093" containerName="test-curl-7266-24ec6b"
Oct 30 22:27:38.040610 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:38.040546    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f4160c3-69cd-4944-92d0-0e5690de1093" containerName="test-curl-7266-24ec6b"
Oct 30 22:27:38.044602 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:38.044585    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7266-c9f195"
Oct 30 22:27:38.059669 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:38.059633    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7266-c9f195"]
Oct 30 22:27:38.091909 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:38.091877    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8prnt\" (UniqueName: \"kubernetes.io/projected/c0d794cd-af67-469a-890d-a868fa24883f-kube-api-access-8prnt\") pod \"test-curl-7266-c9f195\" (UID: \"c0d794cd-af67-469a-890d-a868fa24883f\") " pod="opendatahub/test-curl-7266-c9f195"
Oct 30 22:27:38.193600 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:38.193523    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8prnt\" (UniqueName: \"kubernetes.io/projected/c0d794cd-af67-469a-890d-a868fa24883f-kube-api-access-8prnt\") pod \"test-curl-7266-c9f195\" (UID: \"c0d794cd-af67-469a-890d-a868fa24883f\") " pod="opendatahub/test-curl-7266-c9f195"
Oct 30 22:27:38.213161 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:38.213136    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8prnt\" (UniqueName: \"kubernetes.io/projected/c0d794cd-af67-469a-890d-a868fa24883f-kube-api-access-8prnt\") pod \"test-curl-7266-c9f195\" (UID: \"c0d794cd-af67-469a-890d-a868fa24883f\") " pod="opendatahub/test-curl-7266-c9f195"
Oct 30 22:27:38.388845 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:38.388820    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7266-c9f195"
Oct 30 22:27:38.744284 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:38.736320    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7266-c9f195"]
Oct 30 22:27:38.745866 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:27:38.745823    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0d794cd_af67_469a_890d_a868fa24883f.slice/crio-9f2798231aa76f08b03e5587c31375b7f079c1e5be3509d08c530e8631152d67 WatchSource:0}: Error finding container 9f2798231aa76f08b03e5587c31375b7f079c1e5be3509d08c530e8631152d67: Status 404 returned error can't find the container with id 9f2798231aa76f08b03e5587c31375b7f079c1e5be3509d08c530e8631152d67
Oct 30 22:27:38.771451 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:38.766072    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:38.862077 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:38.857520    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7266-24ec6b" podUID="0f4160c3-69cd-4944-92d0-0e5690de1093" containerName="test-curl-7266-24ec6b" containerID="cri-o://b62b9d2b169586489c2f1af7c3be37a63b3282914e1b03bdb7280a4266d7d774" gracePeriod=2
Oct 30 22:27:38.864115 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:38.864042    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7266-c9f195" event={"ID":"c0d794cd-af67-469a-890d-a868fa24883f","Type":"ContainerStarted","Data":"9f2798231aa76f08b03e5587c31375b7f079c1e5be3509d08c530e8631152d67"}
Oct 30 22:27:38.870096 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:27:38.870060    4889 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7266-24ec6b\" is forbidden: User \"system:node:ip-10-0-142-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-185.ec2.internal' and this object" podUID="0f4160c3-69cd-4944-92d0-0e5690de1093" pod="opendatahub/test-curl-7266-24ec6b"
Oct 30 22:27:39.535305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:39.535157    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:39.901118 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:39.900987    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7266-c9f195" event={"ID":"c0d794cd-af67-469a-890d-a868fa24883f","Type":"ContainerStarted","Data":"a6e156ea1c0c854f638583cc09e0bb2eb92753ba6ad07b98dcafbb0bd846c108"}
Oct 30 22:27:39.938249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:39.936626    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7266-c9f195" podStartSLOduration=1.9366104769999999 podStartE2EDuration="1.936610477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:39.934995522 +0000 UTC m=+1245.291361838" watchObservedRunningTime="2026-10-30 22:27:39.936610477 +0000 UTC m=+1245.292976792"
Oct 30 22:27:40.302271 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:27:40.302216    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:27:40.786742 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:40.786707    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7266-c9f195"]
Oct 30 22:27:40.791745 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:40.791716    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7266-c9f195"]
Oct 30 22:27:40.912328 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:40.912300    4889 generic.go:356] "Generic (PLEG): container finished" podID="0f4160c3-69cd-4944-92d0-0e5690de1093" containerID="b62b9d2b169586489c2f1af7c3be37a63b3282914e1b03bdb7280a4266d7d774" exitCode=137
Oct 30 22:27:40.933876 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:40.933848    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7266-30961e"]
Oct 30 22:27:40.934703 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:40.934682    4889 cpu_manager.go:414] "containerMap: removing container" podUID="c0d794cd-af67-469a-890d-a868fa24883f" containerName="test-curl-7266-c9f195"
Oct 30 22:27:40.934703 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:40.934702    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d794cd-af67-469a-890d-a868fa24883f" containerName="test-curl-7266-c9f195"
Oct 30 22:27:40.934936 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:40.934923    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d794cd-af67-469a-890d-a868fa24883f" containerName="test-curl-7266-c9f195"
Oct 30 22:27:40.938779 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:40.938760    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7266-30961e"
Oct 30 22:27:40.959897 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:40.959864    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7266-30961e"]
Oct 30 22:27:40.986563 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:40.979939    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmbpx\" (UniqueName: \"kubernetes.io/projected/dcc8c725-ebd5-4717-b7d5-2962f4e2a9e7-kube-api-access-nmbpx\") pod \"test-curl-7266-30961e\" (UID: \"dcc8c725-ebd5-4717-b7d5-2962f4e2a9e7\") " pod="opendatahub/test-curl-7266-30961e"
Oct 30 22:27:41.081984 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:41.081955    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmbpx\" (UniqueName: \"kubernetes.io/projected/dcc8c725-ebd5-4717-b7d5-2962f4e2a9e7-kube-api-access-nmbpx\") pod \"test-curl-7266-30961e\" (UID: \"dcc8c725-ebd5-4717-b7d5-2962f4e2a9e7\") " pod="opendatahub/test-curl-7266-30961e"
Oct 30 22:27:41.100097 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:41.100071    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmbpx\" (UniqueName: \"kubernetes.io/projected/dcc8c725-ebd5-4717-b7d5-2962f4e2a9e7-kube-api-access-nmbpx\") pod \"test-curl-7266-30961e\" (UID: \"dcc8c725-ebd5-4717-b7d5-2962f4e2a9e7\") " pod="opendatahub/test-curl-7266-30961e"
Oct 30 22:27:41.132827 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:41.132801    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7266-24ec6b"
Oct 30 22:27:41.183377 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:41.183350    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f4160c3-69cd-4944-92d0-0e5690de1093-kube-api-access-2qxqt\" (UniqueName: \"kubernetes.io/projected/0f4160c3-69cd-4944-92d0-0e5690de1093-kube-api-access-2qxqt\") pod \"0f4160c3-69cd-4944-92d0-0e5690de1093\" (UID: \"0f4160c3-69cd-4944-92d0-0e5690de1093\") "
Oct 30 22:27:41.185615 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:41.185585    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f4160c3-69cd-4944-92d0-0e5690de1093-kube-api-access-2qxqt" pod "0f4160c3-69cd-4944-92d0-0e5690de1093" (UID: "0f4160c3-69cd-4944-92d0-0e5690de1093"). InnerVolumeSpecName "kube-api-access-2qxqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:41.262712 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:41.262666    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7266-30961e"
Oct 30 22:27:41.291314 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:41.286025    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qxqt\" (UniqueName: \"kubernetes.io/projected/0f4160c3-69cd-4944-92d0-0e5690de1093-kube-api-access-2qxqt\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:27:41.360816 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:41.360781    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:41.570368 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:41.559602    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:41.626632 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:41.626602    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7266-30961e"]
Oct 30 22:27:41.629421 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:27:41.629392    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcc8c725_ebd5_4717_b7d5_2962f4e2a9e7.slice/crio-9697585e09558836541e1bd26ded0876775a34d09f7b09c9297b86c4706b628a WatchSource:0}: Error finding container 9697585e09558836541e1bd26ded0876775a34d09f7b09c9297b86c4706b628a: Status 404 returned error can't find the container with id 9697585e09558836541e1bd26ded0876775a34d09f7b09c9297b86c4706b628a
Oct 30 22:27:41.730007 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:41.729978    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:41.921565 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:41.921489    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7266-24ec6b"
Oct 30 22:27:41.921565 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:41.921538    4889 scope.go:122] "RemoveContainer" containerID="b62b9d2b169586489c2f1af7c3be37a63b3282914e1b03bdb7280a4266d7d774"
Oct 30 22:27:41.923673 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:41.923645    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7266-30961e" event={"ID":"dcc8c725-ebd5-4717-b7d5-2962f4e2a9e7","Type":"ContainerStarted","Data":"23c80029e9a268464f390d1dc2ce32f475f348e24b29bf2174bfd87f993beecf"}
Oct 30 22:27:41.923787 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:41.923694    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7266-30961e" event={"ID":"dcc8c725-ebd5-4717-b7d5-2962f4e2a9e7","Type":"ContainerStarted","Data":"9697585e09558836541e1bd26ded0876775a34d09f7b09c9297b86c4706b628a"}
Oct 30 22:27:41.923854 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:41.923799    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7266-c9f195" podUID="c0d794cd-af67-469a-890d-a868fa24883f" containerName="test-curl-7266-c9f195" containerID="cri-o://a6e156ea1c0c854f638583cc09e0bb2eb92753ba6ad07b98dcafbb0bd846c108" gracePeriod=2
Oct 30 22:27:41.961319 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:41.961281    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7266-30961e" podStartSLOduration=1.961269124 podStartE2EDuration="1.961269124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:41.956929235 +0000 UTC m=+1247.313295550" watchObservedRunningTime="2026-10-30 22:27:41.961269124 +0000 UTC m=+1247.317635439"
Oct 30 22:27:41.968975 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:27:41.968947    4889 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7266-c9f195\" is forbidden: User \"system:node:ip-10-0-142-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-185.ec2.internal' and this object" podUID="c0d794cd-af67-469a-890d-a868fa24883f" pod="opendatahub/test-curl-7266-c9f195"
Oct 30 22:27:41.981638 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:27:41.981268    4889 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7266-24ec6b\" is forbidden: User \"system:node:ip-10-0-142-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-185.ec2.internal' and this object" podUID="0f4160c3-69cd-4944-92d0-0e5690de1093" pod="opendatahub/test-curl-7266-24ec6b"
Oct 30 22:27:41.993113 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:27:41.993091    4889 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7266-c9f195\" is forbidden: User \"system:node:ip-10-0-142-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-185.ec2.internal' and this object" podUID="c0d794cd-af67-469a-890d-a868fa24883f" pod="opendatahub/test-curl-7266-c9f195"
Oct 30 22:27:42.784628 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:42.784590    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f4160c3-69cd-4944-92d0-0e5690de1093" path="/var/lib/kubelet/pods/0f4160c3-69cd-4944-92d0-0e5690de1093/volumes"
Oct 30 22:27:42.932502 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:42.932470    4889 generic.go:356] "Generic (PLEG): container finished" podID="d7ce41e7-a1cb-4f49-a247-da6c57d34d53" containerID="f52b786cc3ba6ab73873173d3b67ec58ee39c663f597f780bb9e70b04ae8225a" exitCode=0
Oct 30 22:27:42.932916 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:42.932549    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj" event={"ID":"d7ce41e7-a1cb-4f49-a247-da6c57d34d53","Type":"ContainerDied","Data":"f52b786cc3ba6ab73873173d3b67ec58ee39c663f597f780bb9e70b04ae8225a"}
Oct 30 22:27:42.933083 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:42.933064    4889 scope.go:122] "RemoveContainer" containerID="f52b786cc3ba6ab73873173d3b67ec58ee39c663f597f780bb9e70b04ae8225a"
Oct 30 22:27:42.934961 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:42.934928    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:27:42.935055 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:42.934995    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:27:42.935055 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:42.935033    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:27:42.953129 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:42.951159    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7266-30961e"]
Oct 30 22:27:43.001808 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:42.999116    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7266-30961e"]
Oct 30 22:27:43.120465 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:43.120391    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7266-2826fa"]
Oct 30 22:27:43.121307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:43.121248    4889 cpu_manager.go:414] "containerMap: removing container" podUID="dcc8c725-ebd5-4717-b7d5-2962f4e2a9e7" containerName="test-curl-7266-30961e"
Oct 30 22:27:43.121307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:43.121267    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcc8c725-ebd5-4717-b7d5-2962f4e2a9e7" containerName="test-curl-7266-30961e"
Oct 30 22:27:43.121457 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:43.121446    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcc8c725-ebd5-4717-b7d5-2962f4e2a9e7" containerName="test-curl-7266-30961e"
Oct 30 22:27:43.124470 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:43.124454    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7266-2826fa"
Oct 30 22:27:43.136154 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:43.136124    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7266-2826fa"]
Oct 30 22:27:43.163284 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:43.163139    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxf2k\" (UniqueName: \"kubernetes.io/projected/034371c0-8d3e-4834-b2d2-c57a42a509a9-kube-api-access-gxf2k\") pod \"test-curl-7266-2826fa\" (UID: \"034371c0-8d3e-4834-b2d2-c57a42a509a9\") " pod="opendatahub/test-curl-7266-2826fa"
Oct 30 22:27:43.265071 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:43.265039    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxf2k\" (UniqueName: \"kubernetes.io/projected/034371c0-8d3e-4834-b2d2-c57a42a509a9-kube-api-access-gxf2k\") pod \"test-curl-7266-2826fa\" (UID: \"034371c0-8d3e-4834-b2d2-c57a42a509a9\") " pod="opendatahub/test-curl-7266-2826fa"
Oct 30 22:27:43.285209 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:43.285175    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxf2k\" (UniqueName: \"kubernetes.io/projected/034371c0-8d3e-4834-b2d2-c57a42a509a9-kube-api-access-gxf2k\") pod \"test-curl-7266-2826fa\" (UID: \"034371c0-8d3e-4834-b2d2-c57a42a509a9\") " pod="opendatahub/test-curl-7266-2826fa"
Oct 30 22:27:43.403294 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:43.400979    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:43.451427 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:43.450617    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7266-2826fa"
Oct 30 22:27:43.822555 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:27:43.822527    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod034371c0_8d3e_4834_b2d2_c57a42a509a9.slice/crio-d3a9ab9d98c8453f412196245edc32caab5d0cc66f87b1be6c8431f810a720fb WatchSource:0}: Error finding container d3a9ab9d98c8453f412196245edc32caab5d0cc66f87b1be6c8431f810a720fb: Status 404 returned error can't find the container with id d3a9ab9d98c8453f412196245edc32caab5d0cc66f87b1be6c8431f810a720fb
Oct 30 22:27:43.822954 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:43.822933    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7266-2826fa"]
Oct 30 22:27:44.026268 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:44.025173    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj" event={"ID":"d7ce41e7-a1cb-4f49-a247-da6c57d34d53","Type":"ContainerStarted","Data":"f58f796232e084b576c3ee3532918fcd6300af0b24873b92657698eb0826aac5"}
Oct 30 22:27:44.043430 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:44.042671    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:27:44.043430 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:44.043007    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:27:44.043430 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:44.043055    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj" podUID="d7ce41e7-a1cb-4f49-a247-da6c57d34d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 22:27:44.061208 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:44.060779    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7266-30961e" podUID="dcc8c725-ebd5-4717-b7d5-2962f4e2a9e7" containerName="test-curl-7266-30961e" containerID="cri-o://23c80029e9a268464f390d1dc2ce32f475f348e24b29bf2174bfd87f993beecf" gracePeriod=2
Oct 30 22:27:44.061208 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:44.061132    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7266-2826fa" event={"ID":"034371c0-8d3e-4834-b2d2-c57a42a509a9","Type":"ContainerStarted","Data":"6310755c2c212eab2f39be364a4eebf735ff9f33581ebc832a25f9695817d94c"}
Oct 30 22:27:44.061208 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:44.061169    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7266-2826fa" event={"ID":"034371c0-8d3e-4834-b2d2-c57a42a509a9","Type":"ContainerStarted","Data":"d3a9ab9d98c8453f412196245edc32caab5d0cc66f87b1be6c8431f810a720fb"}
Oct 30 22:27:44.102300 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:27:44.094722    4889 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7266-30961e\" is forbidden: User \"system:node:ip-10-0-142-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-185.ec2.internal' and this object" podUID="dcc8c725-ebd5-4717-b7d5-2962f4e2a9e7" pod="opendatahub/test-curl-7266-30961e"
Oct 30 22:27:44.149092 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:44.149034    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7266-2826fa" podStartSLOduration=1.149016732 podStartE2EDuration="1.149016732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:44.128563563 +0000 UTC m=+1249.484929879" watchObservedRunningTime="2026-10-30 22:27:44.149016732 +0000 UTC m=+1249.505383045"
Oct 30 22:27:44.347451 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:44.347386    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7266-c9f195"
Oct 30 22:27:44.359872 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:27:44.359830    4889 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7266-30961e\" is forbidden: User \"system:node:ip-10-0-142-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-185.ec2.internal' and this object" podUID="dcc8c725-ebd5-4717-b7d5-2962f4e2a9e7" pod="opendatahub/test-curl-7266-30961e"
Oct 30 22:27:44.435350 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:44.435308    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d794cd-af67-469a-890d-a868fa24883f-kube-api-access-8prnt\" (UniqueName: \"kubernetes.io/projected/c0d794cd-af67-469a-890d-a868fa24883f-kube-api-access-8prnt\") pod \"c0d794cd-af67-469a-890d-a868fa24883f\" (UID: \"c0d794cd-af67-469a-890d-a868fa24883f\") "
Oct 30 22:27:44.437784 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:44.437749    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d794cd-af67-469a-890d-a868fa24883f-kube-api-access-8prnt" pod "c0d794cd-af67-469a-890d-a868fa24883f" (UID: "c0d794cd-af67-469a-890d-a868fa24883f"). InnerVolumeSpecName "kube-api-access-8prnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:44.537559 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:44.537520    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8prnt\" (UniqueName: \"kubernetes.io/projected/c0d794cd-af67-469a-890d-a868fa24883f-kube-api-access-8prnt\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:27:44.776564 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:44.776525    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0d794cd-af67-469a-890d-a868fa24883f" path="/var/lib/kubelet/pods/c0d794cd-af67-469a-890d-a868fa24883f/volumes"
Oct 30 22:27:44.783170 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:27:44.783139    4889 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7266-30961e\" is forbidden: User \"system:node:ip-10-0-142-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-185.ec2.internal' and this object" podUID="dcc8c725-ebd5-4717-b7d5-2962f4e2a9e7" pod="opendatahub/test-curl-7266-30961e"
Oct 30 22:27:45.039478 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:45.039378    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:27:45.039478 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:45.039433    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj" podUID="d7ce41e7-a1cb-4f49-a247-da6c57d34d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 22:27:45.046465 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:45.043364    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7266-2826fa"]
Oct 30 22:27:45.047605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:45.047574    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7266-2826fa"]
Oct 30 22:27:45.075601 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:45.075567    4889 generic.go:356] "Generic (PLEG): container finished" podID="c0d794cd-af67-469a-890d-a868fa24883f" containerID="a6e156ea1c0c854f638583cc09e0bb2eb92753ba6ad07b98dcafbb0bd846c108" exitCode=137
Oct 30 22:27:45.075773 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:45.075628    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7266-c9f195"
Oct 30 22:27:45.075773 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:45.075665    4889 scope.go:122] "RemoveContainer" containerID="a6e156ea1c0c854f638583cc09e0bb2eb92753ba6ad07b98dcafbb0bd846c108"
Oct 30 22:27:45.086126 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:45.086110    4889 scope.go:122] "RemoveContainer" containerID="a6e156ea1c0c854f638583cc09e0bb2eb92753ba6ad07b98dcafbb0bd846c108"
Oct 30 22:27:45.086462 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:27:45.086432    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6e156ea1c0c854f638583cc09e0bb2eb92753ba6ad07b98dcafbb0bd846c108\": container with ID starting with a6e156ea1c0c854f638583cc09e0bb2eb92753ba6ad07b98dcafbb0bd846c108 not found: ID does not exist" containerID="a6e156ea1c0c854f638583cc09e0bb2eb92753ba6ad07b98dcafbb0bd846c108"
Oct 30 22:27:45.086564 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:45.086476    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e156ea1c0c854f638583cc09e0bb2eb92753ba6ad07b98dcafbb0bd846c108"} err="failed to get container status \"a6e156ea1c0c854f638583cc09e0bb2eb92753ba6ad07b98dcafbb0bd846c108\": rpc error: code = NotFound desc = could not find container \"a6e156ea1c0c854f638583cc09e0bb2eb92753ba6ad07b98dcafbb0bd846c108\": container with ID starting with a6e156ea1c0c854f638583cc09e0bb2eb92753ba6ad07b98dcafbb0bd846c108 not found: ID does not exist"
Oct 30 22:27:45.331817 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:45.331732    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:45.535199 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:45.535168    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:45.763727 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:45.763695    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:45.937217 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:45.937186    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:46.039584 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:46.039492    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:27:46.040053 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:46.039575    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj" podUID="d7ce41e7-a1cb-4f49-a247-da6c57d34d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 22:27:46.084428 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:46.084395    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7266-2826fa" podUID="034371c0-8d3e-4834-b2d2-c57a42a509a9" containerName="test-curl-7266-2826fa" containerID="cri-o://6310755c2c212eab2f39be364a4eebf735ff9f33581ebc832a25f9695817d94c" gracePeriod=2
Oct 30 22:27:46.096547 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:27:46.096508    4889 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7266-2826fa\" is forbidden: User \"system:node:ip-10-0-142-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-185.ec2.internal' and this object" podUID="034371c0-8d3e-4834-b2d2-c57a42a509a9" pod="opendatahub/test-curl-7266-2826fa"
Oct 30 22:27:46.374873 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:46.374849    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7266-30961e"
Oct 30 22:27:46.387323 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:27:46.387293    4889 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7266-2826fa\" is forbidden: User \"system:node:ip-10-0-142-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-185.ec2.internal' and this object" podUID="034371c0-8d3e-4834-b2d2-c57a42a509a9" pod="opendatahub/test-curl-7266-2826fa"
Oct 30 22:27:46.475571 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:46.475540    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcc8c725-ebd5-4717-b7d5-2962f4e2a9e7-kube-api-access-nmbpx\" (UniqueName: \"kubernetes.io/projected/dcc8c725-ebd5-4717-b7d5-2962f4e2a9e7-kube-api-access-nmbpx\") pod \"dcc8c725-ebd5-4717-b7d5-2962f4e2a9e7\" (UID: \"dcc8c725-ebd5-4717-b7d5-2962f4e2a9e7\") "
Oct 30 22:27:46.478635 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:46.478607    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc8c725-ebd5-4717-b7d5-2962f4e2a9e7-kube-api-access-nmbpx" pod "dcc8c725-ebd5-4717-b7d5-2962f4e2a9e7" (UID: "dcc8c725-ebd5-4717-b7d5-2962f4e2a9e7"). InnerVolumeSpecName "kube-api-access-nmbpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:46.578025 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:46.577988    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmbpx\" (UniqueName: \"kubernetes.io/projected/dcc8c725-ebd5-4717-b7d5-2962f4e2a9e7-kube-api-access-nmbpx\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:27:46.778646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:46.778613    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcc8c725-ebd5-4717-b7d5-2962f4e2a9e7" path="/var/lib/kubelet/pods/dcc8c725-ebd5-4717-b7d5-2962f4e2a9e7/volumes"
Oct 30 22:27:46.936994 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:46.936950    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:47.040679 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:47.040597    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:27:47.040679 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:47.040644    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj" podUID="d7ce41e7-a1cb-4f49-a247-da6c57d34d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 22:27:47.092695 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:47.092661    4889 generic.go:356] "Generic (PLEG): container finished" podID="dcc8c725-ebd5-4717-b7d5-2962f4e2a9e7" containerID="23c80029e9a268464f390d1dc2ce32f475f348e24b29bf2174bfd87f993beecf" exitCode=137
Oct 30 22:27:47.092863 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:47.092709    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7266-30961e"
Oct 30 22:27:47.092863 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:47.092761    4889 scope.go:122] "RemoveContainer" containerID="23c80029e9a268464f390d1dc2ce32f475f348e24b29bf2174bfd87f993beecf"
Oct 30 22:27:47.103785 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:47.103765    4889 scope.go:122] "RemoveContainer" containerID="23c80029e9a268464f390d1dc2ce32f475f348e24b29bf2174bfd87f993beecf"
Oct 30 22:27:47.104084 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:27:47.104065    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23c80029e9a268464f390d1dc2ce32f475f348e24b29bf2174bfd87f993beecf\": container with ID starting with 23c80029e9a268464f390d1dc2ce32f475f348e24b29bf2174bfd87f993beecf not found: ID does not exist" containerID="23c80029e9a268464f390d1dc2ce32f475f348e24b29bf2174bfd87f993beecf"
Oct 30 22:27:47.104139 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:47.104092    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23c80029e9a268464f390d1dc2ce32f475f348e24b29bf2174bfd87f993beecf"} err="failed to get container status \"23c80029e9a268464f390d1dc2ce32f475f348e24b29bf2174bfd87f993beecf\": rpc error: code = NotFound desc = could not find container \"23c80029e9a268464f390d1dc2ce32f475f348e24b29bf2174bfd87f993beecf\": container with ID starting with 23c80029e9a268464f390d1dc2ce32f475f348e24b29bf2174bfd87f993beecf not found: ID does not exist"
Oct 30 22:27:47.137282 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:47.134312    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:47.336385 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:47.336211    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:48.039300 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:48.039261    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:27:48.039501 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:48.039324    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj" podUID="d7ce41e7-a1cb-4f49-a247-da6c57d34d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 22:27:48.384875 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:48.384848    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7266-2826fa"
Oct 30 22:27:48.545011 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:48.544976    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/034371c0-8d3e-4834-b2d2-c57a42a509a9-kube-api-access-gxf2k\" (UniqueName: \"kubernetes.io/projected/034371c0-8d3e-4834-b2d2-c57a42a509a9-kube-api-access-gxf2k\") pod \"034371c0-8d3e-4834-b2d2-c57a42a509a9\" (UID: \"034371c0-8d3e-4834-b2d2-c57a42a509a9\") "
Oct 30 22:27:48.547316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:48.547291    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/034371c0-8d3e-4834-b2d2-c57a42a509a9-kube-api-access-gxf2k" pod "034371c0-8d3e-4834-b2d2-c57a42a509a9" (UID: "034371c0-8d3e-4834-b2d2-c57a42a509a9"). InnerVolumeSpecName "kube-api-access-gxf2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:48.647763 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:48.647574    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxf2k\" (UniqueName: \"kubernetes.io/projected/034371c0-8d3e-4834-b2d2-c57a42a509a9-kube-api-access-gxf2k\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:27:48.776063 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:48.776031    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="034371c0-8d3e-4834-b2d2-c57a42a509a9" path="/var/lib/kubelet/pods/034371c0-8d3e-4834-b2d2-c57a42a509a9/volumes"
Oct 30 22:27:49.038975 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:49.038939    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:27:49.039338 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:49.039303    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj" podUID="d7ce41e7-a1cb-4f49-a247-da6c57d34d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 22:27:49.070247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:49.065002    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:49.153250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:49.145806    4889 generic.go:356] "Generic (PLEG): container finished" podID="034371c0-8d3e-4834-b2d2-c57a42a509a9" containerID="6310755c2c212eab2f39be364a4eebf735ff9f33581ebc832a25f9695817d94c" exitCode=137
Oct 30 22:27:49.153250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:49.145903    4889 scope.go:122] "RemoveContainer" containerID="6310755c2c212eab2f39be364a4eebf735ff9f33581ebc832a25f9695817d94c"
Oct 30 22:27:49.153250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:49.146103    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7266-2826fa"
Oct 30 22:27:49.201703 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:49.193087    4889 scope.go:122] "RemoveContainer" containerID="6310755c2c212eab2f39be364a4eebf735ff9f33581ebc832a25f9695817d94c"
Oct 30 22:27:49.201703 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:27:49.193477    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6310755c2c212eab2f39be364a4eebf735ff9f33581ebc832a25f9695817d94c\": container with ID starting with 6310755c2c212eab2f39be364a4eebf735ff9f33581ebc832a25f9695817d94c not found: ID does not exist" containerID="6310755c2c212eab2f39be364a4eebf735ff9f33581ebc832a25f9695817d94c"
Oct 30 22:27:49.201703 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:49.193507    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6310755c2c212eab2f39be364a4eebf735ff9f33581ebc832a25f9695817d94c"} err="failed to get container status \"6310755c2c212eab2f39be364a4eebf735ff9f33581ebc832a25f9695817d94c\": rpc error: code = NotFound desc = could not find container \"6310755c2c212eab2f39be364a4eebf735ff9f33581ebc832a25f9695817d94c\": container with ID starting with 6310755c2c212eab2f39be364a4eebf735ff9f33581ebc832a25f9695817d94c not found: ID does not exist"
Oct 30 22:27:49.707687 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:49.707660    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:27:50.043538 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:50.043509    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:27:50.156407 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:50.156381    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 22:27:50.841054 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:50.840975    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:52.780941 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:52.780900    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:53.277344 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:53.277313    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:54.668290 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:54.666899    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:56.357101 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:56.357065    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:56.600822 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:56.600784    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:56.923932 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:56.923883    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:58.170610 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:58.170576    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:59.831212 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:27:59.831163    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:01.529741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:01.529704    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:03.191245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:03.185980    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:04.835784 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:04.835750    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:07.843249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:07.842785    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:10.152248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:10.134094    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:10.484655 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:28:10.484453    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:28:11.746517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:11.746479    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:12.009751 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:12.009659    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:12.634374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:12.634338    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:12.811441 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:12.811408    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:13.332830 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:13.332798    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:14.052125 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:14.050854    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:14.817543 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:14.817509    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:15.316808 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:15.316777    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:15.779579 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:15.779544    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:15.956245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:15.956201    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:17.656246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:17.655845    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:18.262052 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:18.262005    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:19.282439 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:19.282404    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7278-c7a0ef"]
Oct 30 22:28:19.283177 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:19.283159    4889 cpu_manager.go:414] "containerMap: removing container" podUID="034371c0-8d3e-4834-b2d2-c57a42a509a9" containerName="test-curl-7266-2826fa"
Oct 30 22:28:19.283177 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:19.283176    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="034371c0-8d3e-4834-b2d2-c57a42a509a9" containerName="test-curl-7266-2826fa"
Oct 30 22:28:19.283446 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:19.283429    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="034371c0-8d3e-4834-b2d2-c57a42a509a9" containerName="test-curl-7266-2826fa"
Oct 30 22:28:19.287430 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:19.287413    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-c7a0ef"
Oct 30 22:28:19.302050 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:19.302025    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7278-c7a0ef"]
Oct 30 22:28:19.306839 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:19.306816    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xbw52\""
Oct 30 22:28:19.456035 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:19.455605    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:19.464547 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:19.464499    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpkbh\" (UniqueName: \"kubernetes.io/projected/6ef29f10-108d-4e14-a445-37dea11b718f-kube-api-access-xpkbh\") pod \"test-curl-7278-c7a0ef\" (UID: \"6ef29f10-108d-4e14-a445-37dea11b718f\") " pod="opendatahub/test-curl-7278-c7a0ef"
Oct 30 22:28:19.574353 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:19.574265    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpkbh\" (UniqueName: \"kubernetes.io/projected/6ef29f10-108d-4e14-a445-37dea11b718f-kube-api-access-xpkbh\") pod \"test-curl-7278-c7a0ef\" (UID: \"6ef29f10-108d-4e14-a445-37dea11b718f\") " pod="opendatahub/test-curl-7278-c7a0ef"
Oct 30 22:28:19.594976 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:19.594945    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpkbh\" (UniqueName: \"kubernetes.io/projected/6ef29f10-108d-4e14-a445-37dea11b718f-kube-api-access-xpkbh\") pod \"test-curl-7278-c7a0ef\" (UID: \"6ef29f10-108d-4e14-a445-37dea11b718f\") " pod="opendatahub/test-curl-7278-c7a0ef"
Oct 30 22:28:19.600889 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:19.600853    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-c7a0ef"
Oct 30 22:28:19.901192 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:19.901149    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7278-c7a0ef"]
Oct 30 22:28:19.901421 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:28:19.901395    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ef29f10_108d_4e14_a445_37dea11b718f.slice/crio-9f584c46b715333d4ced7e41c7a6ebcf6ffdeb5d0ac53056b25d729108d9e5cd WatchSource:0}: Error finding container 9f584c46b715333d4ced7e41c7a6ebcf6ffdeb5d0ac53056b25d729108d9e5cd: Status 404 returned error can't find the container with id 9f584c46b715333d4ced7e41c7a6ebcf6ffdeb5d0ac53056b25d729108d9e5cd
Oct 30 22:28:19.948078 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:19.948039    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:20.118763 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:20.118729    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:20.291837 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:20.291804    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:20.608550 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:20.608455    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7278-c7a0ef" event={"ID":"6ef29f10-108d-4e14-a445-37dea11b718f","Type":"ContainerStarted","Data":"c2bd7d30dccdc6567b2bf7fa4bbbea384cf1865143e606463bf2ceb22e71a92d"}
Oct 30 22:28:20.608550 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:20.608506    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7278-c7a0ef" event={"ID":"6ef29f10-108d-4e14-a445-37dea11b718f","Type":"ContainerStarted","Data":"9f584c46b715333d4ced7e41c7a6ebcf6ffdeb5d0ac53056b25d729108d9e5cd"}
Oct 30 22:28:20.645310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:20.645262    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7278-c7a0ef" podStartSLOduration=1.645247965 podStartE2EDuration="1.645247965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:20.643483446 +0000 UTC m=+1285.999849761" watchObservedRunningTime="2026-10-30 22:28:20.645247965 +0000 UTC m=+1286.001614270"
Oct 30 22:28:21.457101 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.455421    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:21.583287 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.582752    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7278-c7a0ef"]
Oct 30 22:28:21.587990 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.587902    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7278-c7a0ef"]
Oct 30 22:28:21.615036 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.613098    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"]
Oct 30 22:28:21.615036 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.614219    4889 cpu_manager.go:414] "containerMap: removing container" podUID="6ef29f10-108d-4e14-a445-37dea11b718f" containerName="test-curl-7278-c7a0ef"
Oct 30 22:28:21.615036 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.614257    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef29f10-108d-4e14-a445-37dea11b718f" containerName="test-curl-7278-c7a0ef"
Oct 30 22:28:21.615036 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.614434    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef29f10-108d-4e14-a445-37dea11b718f" containerName="test-curl-7278-c7a0ef"
Oct 30 22:28:21.618877 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.618857    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.642658 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.642618    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"]
Oct 30 22:28:21.646359 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.646330    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d399fb07-openshift-default-dockercfg-gkn9c\""
Oct 30 22:28:21.715763 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.715672    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-podinfo\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.715763 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.715730    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istiod-ca-cert\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.715997 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.715794    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9p94\" (UniqueName: \"kubernetes.io/projected/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-kube-api-access-l9p94\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.715997 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.715839    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-workload-socket\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.715997 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.715888    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-data\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.715997 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.715968    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-envoy\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.716201 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.716117    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-credential-socket\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.716400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.716268    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-workload-certs\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.716400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.716312    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-token\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.759042 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.759010    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:21.818155 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.818126    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istiod-ca-cert\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.818374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.818174    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9p94\" (UniqueName: \"kubernetes.io/projected/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-kube-api-access-l9p94\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.818374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.818202    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-workload-socket\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.818374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.818244    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-data\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.818374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.818288    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-envoy\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.818374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.818364    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-credential-socket\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.818756 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.818452    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-workload-certs\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.818756 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.818485    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-token\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.818756 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.818588    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-podinfo\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.818756 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.818648    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-workload-socket\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.818981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.818961    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istiod-ca-cert\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.818981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.818967    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-data\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.819086 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.819029    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-workload-certs\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.819185 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.819164    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-credential-socket\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.821197 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.821171    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-envoy\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.821341 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.821324    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-podinfo\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.836316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.836295    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9p94\" (UniqueName: \"kubernetes.io/projected/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-kube-api-access-l9p94\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.836425 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.836378    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-token\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.881335 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.881301    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995"]
Oct 30 22:28:21.887608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.887586    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995"
Oct 30 22:28:21.901063 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.901038    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-510f80-dockercfg-ctrd4\""
Oct 30 22:28:21.901166 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.901099    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-510f80\""
Oct 30 22:28:21.903077 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.903053    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995"]
Oct 30 22:28:21.920032 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.920003    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqznk\" (UniqueName: \"kubernetes.io/projected/e0a01454-5acd-417b-9e3c-62c826ef6558-kube-api-access-gqznk\") pod \"payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995\" (UID: \"e0a01454-5acd-417b-9e3c-62c826ef6558\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995"
Oct 30 22:28:21.920217 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.920192    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a01454-5acd-417b-9e3c-62c826ef6558-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995\" (UID: \"e0a01454-5acd-417b-9e3c-62c826ef6558\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995"
Oct 30 22:28:21.949318 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.943960    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:21.949318 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:21.947171    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:22.039250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:22.039027    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqznk\" (UniqueName: \"kubernetes.io/projected/e0a01454-5acd-417b-9e3c-62c826ef6558-kube-api-access-gqznk\") pod \"payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995\" (UID: \"e0a01454-5acd-417b-9e3c-62c826ef6558\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995"
Oct 30 22:28:22.039452 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:22.039257    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a01454-5acd-417b-9e3c-62c826ef6558-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995\" (UID: \"e0a01454-5acd-417b-9e3c-62c826ef6558\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995"
Oct 30 22:28:22.069667 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:22.062526    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a01454-5acd-417b-9e3c-62c826ef6558-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995\" (UID: \"e0a01454-5acd-417b-9e3c-62c826ef6558\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995"
Oct 30 22:28:22.434038 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:22.426543    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqznk\" (UniqueName: \"kubernetes.io/projected/e0a01454-5acd-417b-9e3c-62c826ef6558-kube-api-access-gqznk\") pod \"payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995\" (UID: \"e0a01454-5acd-417b-9e3c-62c826ef6558\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995"
Oct 30 22:28:22.449321 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:22.439837    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm"]
Oct 30 22:28:22.494095 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:22.494064    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm"]
Oct 30 22:28:22.494778 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:22.494760    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm"
Oct 30 22:28:22.505813 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:22.505405    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995"
Oct 30 22:28:22.539183 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:22.539140    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9wtz\" (UniqueName: \"kubernetes.io/projected/2ace3d4c-11b6-41ca-a276-37746b3791fa-kube-api-access-t9wtz\") pod \"payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm\" (UID: \"2ace3d4c-11b6-41ca-a276-37746b3791fa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm"
Oct 30 22:28:22.539365 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:22.539273    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ace3d4c-11b6-41ca-a276-37746b3791fa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm\" (UID: \"2ace3d4c-11b6-41ca-a276-37746b3791fa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm"
Oct 30 22:28:22.629069 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:22.629033    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7278-c7a0ef" podUID="6ef29f10-108d-4e14-a445-37dea11b718f" containerName="test-curl-7278-c7a0ef" containerID="cri-o://c2bd7d30dccdc6567b2bf7fa4bbbea384cf1865143e606463bf2ceb22e71a92d" gracePeriod=2
Oct 30 22:28:22.640423 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:22.640394    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ace3d4c-11b6-41ca-a276-37746b3791fa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm\" (UID: \"2ace3d4c-11b6-41ca-a276-37746b3791fa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm"
Oct 30 22:28:22.640743 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:22.640721    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9wtz\" (UniqueName: \"kubernetes.io/projected/2ace3d4c-11b6-41ca-a276-37746b3791fa-kube-api-access-t9wtz\") pod \"payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm\" (UID: \"2ace3d4c-11b6-41ca-a276-37746b3791fa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm"
Oct 30 22:28:22.641166 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:22.641148    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ace3d4c-11b6-41ca-a276-37746b3791fa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm\" (UID: \"2ace3d4c-11b6-41ca-a276-37746b3791fa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm"
Oct 30 22:28:22.641821 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:28:22.641795    4889 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7278-c7a0ef\" is forbidden: User \"system:node:ip-10-0-142-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-185.ec2.internal' and this object" podUID="6ef29f10-108d-4e14-a445-37dea11b718f" pod="opendatahub/test-curl-7278-c7a0ef"
Oct 30 22:28:22.658868 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:22.658844    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9wtz\" (UniqueName: \"kubernetes.io/projected/2ace3d4c-11b6-41ca-a276-37746b3791fa-kube-api-access-t9wtz\") pod \"payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm\" (UID: \"2ace3d4c-11b6-41ca-a276-37746b3791fa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm"
Oct 30 22:28:22.705722 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:22.705693    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"]
Oct 30 22:28:22.706113 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:28:22.706093    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0ab0bb_35e8_4e4e_8f79_35cfd6e56820.slice/crio-ec58b242c96ee02e9f79b77a41fd3dc1e1b1a5aa239cb26cb3b3badef2c5ebbc WatchSource:0}: Error finding container ec58b242c96ee02e9f79b77a41fd3dc1e1b1a5aa239cb26cb3b3badef2c5ebbc: Status 404 returned error can't find the container with id ec58b242c96ee02e9f79b77a41fd3dc1e1b1a5aa239cb26cb3b3badef2c5ebbc
Oct 30 22:28:22.708930 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:22.708897    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:28:22.709016 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:22.708963    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:28:22.709016 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:22.708994    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:28:22.832873 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:22.832842    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995"]
Oct 30 22:28:22.833120 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:22.833098    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm"
Oct 30 22:28:23.155420 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:23.155338    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm"]
Oct 30 22:28:23.681535 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:23.681437    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:23.746260 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:23.742139    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm" event={"ID":"2ace3d4c-11b6-41ca-a276-37746b3791fa","Type":"ContainerStarted","Data":"e42435c4b1d5891e13096494a67eb64caeb516f387614131dfd299be6d59616d"}
Oct 30 22:28:23.746260 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:23.742206    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm" event={"ID":"2ace3d4c-11b6-41ca-a276-37746b3791fa","Type":"ContainerStarted","Data":"80ed7084784c5c5c9f3ab78a1b70a107d3069d9a88f250efb066ad4c1af29ce4"}
Oct 30 22:28:23.747950 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:23.747924    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm"
Oct 30 22:28:23.784325 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:23.779212    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995" event={"ID":"e0a01454-5acd-417b-9e3c-62c826ef6558","Type":"ContainerStarted","Data":"fcb7050194b24fe2cfd5e1eda326c9a02a60a76014c202a89a1d592c48fcd3dd"}
Oct 30 22:28:23.784325 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:23.779306    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995" event={"ID":"e0a01454-5acd-417b-9e3c-62c826ef6558","Type":"ContainerStarted","Data":"8808f44905c89c01e477c697d02a2842435024c1dc58d29788bbcfd2ff5a4e55"}
Oct 30 22:28:23.790540 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:23.780210    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995"
Oct 30 22:28:23.805856 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:23.804954    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm" podStartSLOduration=2.804936507 podStartE2EDuration="2.804936507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:23.7908287 +0000 UTC m=+1289.147195048" watchObservedRunningTime="2026-10-30 22:28:23.804936507 +0000 UTC m=+1289.161302811"
Oct 30 22:28:23.818210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:23.817282    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9" event={"ID":"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820","Type":"ContainerStarted","Data":"2cbebb70dd8923a6a9ab9154a77f16b682f5f712df5a4141d784ba63a03be1e8"}
Oct 30 22:28:23.818210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:23.817340    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9" event={"ID":"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820","Type":"ContainerStarted","Data":"ec58b242c96ee02e9f79b77a41fd3dc1e1b1a5aa239cb26cb3b3badef2c5ebbc"}
Oct 30 22:28:23.839381 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:23.839309    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995" podStartSLOduration=2.839289365 podStartE2EDuration="2.839289365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:23.837805936 +0000 UTC m=+1289.194172250" watchObservedRunningTime="2026-10-30 22:28:23.839289365 +0000 UTC m=+1289.195655680"
Oct 30 22:28:23.882783 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:23.882716    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9" podStartSLOduration=2.882695884 podStartE2EDuration="2.882695884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:23.877541238 +0000 UTC m=+1289.233907552" watchObservedRunningTime="2026-10-30 22:28:23.882695884 +0000 UTC m=+1289.239062241"
Oct 30 22:28:23.947457 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:23.947294    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:23.949182 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:23.949154    4889 patch_prober.go:29] interesting pod/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:28:23.949571 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:23.949498    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9" podUID="eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 22:28:24.873317 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:24.873283    4889 generic.go:356] "Generic (PLEG): container finished" podID="6ef29f10-108d-4e14-a445-37dea11b718f" containerID="c2bd7d30dccdc6567b2bf7fa4bbbea384cf1865143e606463bf2ceb22e71a92d" exitCode=137
Oct 30 22:28:24.947555 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:24.947523    4889 patch_prober.go:29] interesting pod/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:28:24.947683 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:24.947587    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9" podUID="eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 22:28:24.973106 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:24.973078    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-c7a0ef"
Oct 30 22:28:24.985370 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:28:24.985337    4889 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7278-c7a0ef\" is forbidden: User \"system:node:ip-10-0-142-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-185.ec2.internal' and this object" podUID="6ef29f10-108d-4e14-a445-37dea11b718f" pod="opendatahub/test-curl-7278-c7a0ef"
Oct 30 22:28:25.069768 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:25.069738    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef29f10-108d-4e14-a445-37dea11b718f-kube-api-access-xpkbh\" (UniqueName: \"kubernetes.io/projected/6ef29f10-108d-4e14-a445-37dea11b718f-kube-api-access-xpkbh\") pod \"6ef29f10-108d-4e14-a445-37dea11b718f\" (UID: \"6ef29f10-108d-4e14-a445-37dea11b718f\") "
Oct 30 22:28:25.071998 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:25.071970    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef29f10-108d-4e14-a445-37dea11b718f-kube-api-access-xpkbh" pod "6ef29f10-108d-4e14-a445-37dea11b718f" (UID: "6ef29f10-108d-4e14-a445-37dea11b718f"). InnerVolumeSpecName "kube-api-access-xpkbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:25.171961 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:25.171929    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpkbh\" (UniqueName: \"kubernetes.io/projected/6ef29f10-108d-4e14-a445-37dea11b718f-kube-api-access-xpkbh\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:28:25.767266 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:25.767009    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:25.882007 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:25.881974    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-c7a0ef"
Oct 30 22:28:25.882007 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:25.882006    4889 scope.go:122] "RemoveContainer" containerID="c2bd7d30dccdc6567b2bf7fa4bbbea384cf1865143e606463bf2ceb22e71a92d"
Oct 30 22:28:25.894857 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:28:25.894672    4889 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7278-c7a0ef\" is forbidden: User \"system:node:ip-10-0-142-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-185.ec2.internal' and this object" podUID="6ef29f10-108d-4e14-a445-37dea11b718f" pod="opendatahub/test-curl-7278-c7a0ef"
Oct 30 22:28:25.907186 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:28:25.907157    4889 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7278-c7a0ef\" is forbidden: User \"system:node:ip-10-0-142-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-185.ec2.internal' and this object" podUID="6ef29f10-108d-4e14-a445-37dea11b718f" pod="opendatahub/test-curl-7278-c7a0ef"
Oct 30 22:28:25.947634 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:25.947603    4889 patch_prober.go:29] interesting pod/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:28:25.947772 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:25.947662    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9" podUID="eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 22:28:26.072652 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:26.072563    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:26.778096 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:26.778065    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ef29f10-108d-4e14-a445-37dea11b718f" path="/var/lib/kubelet/pods/6ef29f10-108d-4e14-a445-37dea11b718f/volumes"
Oct 30 22:28:26.947215 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:26.947184    4889 patch_prober.go:29] interesting pod/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:28:26.947640 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:26.947250    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9" podUID="eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 22:28:27.711245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:27.703765    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:27.946940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:27.946910    4889 patch_prober.go:29] interesting pod/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:28:27.947103 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:27.946962    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9" podUID="eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 22:28:28.947155 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:28.947120    4889 patch_prober.go:29] interesting pod/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:28:28.947590 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:28.947176    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9" podUID="eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 22:28:29.582430 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:29.581279    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:29.952304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:29.952180    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:29.952708 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:29.952518    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:29.953337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:29.953317    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:28:31.294255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:31.285690    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:32.778539 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:32.778509    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7"]
Oct 30 22:28:32.807583 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:32.807552    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt"]
Oct 30 22:28:32.967098 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:32.967062    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:33.216794 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:33.216640    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:34.789309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:34.783724    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:35.884886 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:35.884858    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995"
Oct 30 22:28:35.885355 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:35.884999    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm"
Oct 30 22:28:36.550282 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:36.542608    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:37.779682 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:37.779644    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7" podUID="e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29" containerName="payload-processing" containerID="cri-o://81a56a08a6c26da7b262052180d5c3babebb44e8929d746683018382d74ea536" gracePeriod=25
Oct 30 22:28:37.808365 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:37.808336    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt" podUID="800dedce-28ae-4996-ac41-0253054aea5a" containerName="payload-pre-processing" containerID="cri-o://138e6e5f9bfb614a72240554aad30eec86b260229fabb74e65a384cb54098662" gracePeriod=25
Oct 30 22:28:38.298962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:38.298922    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:40.025249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:40.021115    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:40.303301 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:28:40.303203    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:28:40.789449 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:40.789405    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt" podUID="800dedce-28ae-4996-ac41-0253054aea5a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:28:40.789675 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:40.789651    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7" podUID="e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:28:41.727248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:41.726094    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:43.416667 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:43.416629    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:45.102030 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:45.101996    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:46.817078 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:46.817045    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:47.105961 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:47.105879    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:47.303012 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:47.302978    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:47.497672 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:47.497641    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:48.889211 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:48.889175    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:50.722947 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:50.722907    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:50.793723 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:50.793672    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7" podUID="e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:28:50.795964 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:50.795875    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt" podUID="800dedce-28ae-4996-ac41-0253054aea5a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:28:52.469270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:52.463436    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:54.230251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:54.224264    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:55.959459 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:55.959420    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:57.726466 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:57.726427    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:58.112315 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.112287    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7"
Oct 30 22:28:58.166491 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.166459    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt"
Oct 30 22:28:58.228290 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.228262    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29-kube-api-access-6htrf\" (UniqueName: \"kubernetes.io/projected/e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29-kube-api-access-6htrf\") pod \"e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29\" (UID: \"e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29\") "
Oct 30 22:28:58.228452 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.228422    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29-plugins-config-volume\") pod \"e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29\" (UID: \"e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29\") "
Oct 30 22:28:58.228791 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.228765    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29-plugins-config-volume" pod "e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29" (UID: "e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:58.229130 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.229111    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:28:58.230755 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.230695    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29-kube-api-access-6htrf" pod "e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29" (UID: "e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29"). InnerVolumeSpecName "kube-api-access-6htrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:58.329744 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.329711    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/800dedce-28ae-4996-ac41-0253054aea5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/800dedce-28ae-4996-ac41-0253054aea5a-plugins-config-volume\") pod \"800dedce-28ae-4996-ac41-0253054aea5a\" (UID: \"800dedce-28ae-4996-ac41-0253054aea5a\") "
Oct 30 22:28:58.329906 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.329889    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/800dedce-28ae-4996-ac41-0253054aea5a-kube-api-access-l8fmh\" (UniqueName: \"kubernetes.io/projected/800dedce-28ae-4996-ac41-0253054aea5a-kube-api-access-l8fmh\") pod \"800dedce-28ae-4996-ac41-0253054aea5a\" (UID: \"800dedce-28ae-4996-ac41-0253054aea5a\") "
Oct 30 22:28:58.330076 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.330056    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/800dedce-28ae-4996-ac41-0253054aea5a-plugins-config-volume" pod "800dedce-28ae-4996-ac41-0253054aea5a" (UID: "800dedce-28ae-4996-ac41-0253054aea5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:58.330626 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.330606    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/800dedce-28ae-4996-ac41-0253054aea5a-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:28:58.330626 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.330625    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6htrf\" (UniqueName: \"kubernetes.io/projected/e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29-kube-api-access-6htrf\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:28:58.332044 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.332023    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/800dedce-28ae-4996-ac41-0253054aea5a-kube-api-access-l8fmh" pod "800dedce-28ae-4996-ac41-0253054aea5a" (UID: "800dedce-28ae-4996-ac41-0253054aea5a"). InnerVolumeSpecName "kube-api-access-l8fmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:58.341336 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.341312    4889 generic.go:356] "Generic (PLEG): container finished" podID="800dedce-28ae-4996-ac41-0253054aea5a" containerID="138e6e5f9bfb614a72240554aad30eec86b260229fabb74e65a384cb54098662" exitCode=0
Oct 30 22:28:58.341466 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.341365    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt"
Oct 30 22:28:58.341466 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.341392    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt" event={"ID":"800dedce-28ae-4996-ac41-0253054aea5a","Type":"ContainerDied","Data":"138e6e5f9bfb614a72240554aad30eec86b260229fabb74e65a384cb54098662"}
Oct 30 22:28:58.341571 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.341468    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt" event={"ID":"800dedce-28ae-4996-ac41-0253054aea5a","Type":"ContainerDied","Data":"83c3745c1756026f9d5e8511559d22154754ed80eeafd7c2afee564c553986e5"}
Oct 30 22:28:58.341571 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.341499    4889 scope.go:122] "RemoveContainer" containerID="138e6e5f9bfb614a72240554aad30eec86b260229fabb74e65a384cb54098662"
Oct 30 22:28:58.342965 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.342937    4889 generic.go:356] "Generic (PLEG): container finished" podID="e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29" containerID="81a56a08a6c26da7b262052180d5c3babebb44e8929d746683018382d74ea536" exitCode=0
Oct 30 22:28:58.343072 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.342974    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7"
Oct 30 22:28:58.343072 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.343039    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7" event={"ID":"e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29","Type":"ContainerDied","Data":"81a56a08a6c26da7b262052180d5c3babebb44e8929d746683018382d74ea536"}
Oct 30 22:28:58.343174 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.343087    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7" event={"ID":"e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29","Type":"ContainerDied","Data":"0732acc34f5561468113cf64834a4497995105058754e0f9e3d26397a886aa6b"}
Oct 30 22:28:58.353459 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.353440    4889 scope.go:122] "RemoveContainer" containerID="138e6e5f9bfb614a72240554aad30eec86b260229fabb74e65a384cb54098662"
Oct 30 22:28:58.353734 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:28:58.353712    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"138e6e5f9bfb614a72240554aad30eec86b260229fabb74e65a384cb54098662\": container with ID starting with 138e6e5f9bfb614a72240554aad30eec86b260229fabb74e65a384cb54098662 not found: ID does not exist" containerID="138e6e5f9bfb614a72240554aad30eec86b260229fabb74e65a384cb54098662"
Oct 30 22:28:58.353789 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.353743    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"138e6e5f9bfb614a72240554aad30eec86b260229fabb74e65a384cb54098662"} err="failed to get container status \"138e6e5f9bfb614a72240554aad30eec86b260229fabb74e65a384cb54098662\": rpc error: code = NotFound desc = could not find container \"138e6e5f9bfb614a72240554aad30eec86b260229fabb74e65a384cb54098662\": container with ID starting with 138e6e5f9bfb614a72240554aad30eec86b260229fabb74e65a384cb54098662 not found: ID does not exist"
Oct 30 22:28:58.353789 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.353760    4889 scope.go:122] "RemoveContainer" containerID="81a56a08a6c26da7b262052180d5c3babebb44e8929d746683018382d74ea536"
Oct 30 22:28:58.363747 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.363728    4889 scope.go:122] "RemoveContainer" containerID="81a56a08a6c26da7b262052180d5c3babebb44e8929d746683018382d74ea536"
Oct 30 22:28:58.364014 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:28:58.363998    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81a56a08a6c26da7b262052180d5c3babebb44e8929d746683018382d74ea536\": container with ID starting with 81a56a08a6c26da7b262052180d5c3babebb44e8929d746683018382d74ea536 not found: ID does not exist" containerID="81a56a08a6c26da7b262052180d5c3babebb44e8929d746683018382d74ea536"
Oct 30 22:28:58.364058 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.364020    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81a56a08a6c26da7b262052180d5c3babebb44e8929d746683018382d74ea536"} err="failed to get container status \"81a56a08a6c26da7b262052180d5c3babebb44e8929d746683018382d74ea536\": rpc error: code = NotFound desc = could not find container \"81a56a08a6c26da7b262052180d5c3babebb44e8929d746683018382d74ea536\": container with ID starting with 81a56a08a6c26da7b262052180d5c3babebb44e8929d746683018382d74ea536 not found: ID does not exist"
Oct 30 22:28:58.392278 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.392255    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt"]
Oct 30 22:28:58.396546 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.396515    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-6qcmt"]
Oct 30 22:28:58.426093 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.426047    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7"]
Oct 30 22:28:58.430411 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.430390    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jm8q7"]
Oct 30 22:28:58.432260 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.432238    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8fmh\" (UniqueName: \"kubernetes.io/projected/800dedce-28ae-4996-ac41-0253054aea5a-kube-api-access-l8fmh\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:28:58.777009 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.776980    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="800dedce-28ae-4996-ac41-0253054aea5a" path="/var/lib/kubelet/pods/800dedce-28ae-4996-ac41-0253054aea5a/volumes"
Oct 30 22:28:58.777460 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:58.777429    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29" path="/var/lib/kubelet/pods/e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29/volumes"
Oct 30 22:28:59.538799 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:28:59.538764    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:01.326395 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:01.326351    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:03.087270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:03.087169    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:04.798719 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:04.798669    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:06.628053 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:06.628012    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:08.545743 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:08.545704    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:08.866132 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:08.866038    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"]
Oct 30 22:29:08.866425 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:08.866242    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k" podUID="4f63c047-c985-45ed-92b5-3fed61c84385" containerName="istio-proxy" containerID="cri-o://967d05a6cea8cfdee6de2d406efa400eae39299234f916036c8e6f633b13d631" gracePeriod=30
Oct 30 22:29:08.877965 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:08.877936    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"]
Oct 30 22:29:08.878199 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:08.878177    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8" podUID="08eb9eea-09b6-4149-9c58-5a252ef3062a" containerName="istio-proxy" containerID="cri-o://802fd5d0b6f4a8a2dd4205d5e826ce855238318dc56569db2cfac5f641f8d94d" gracePeriod=30
Oct 30 22:29:08.883774 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:08.883744    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"]
Oct 30 22:29:08.885381 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:08.885346    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv" podUID="408d956d-ee1c-44e8-9bc5-8d3fc72ab890" containerName="istio-proxy" containerID="cri-o://d42888dc1dab5f808a283a03d24fc1c473662a6fcecf7d7ccf3d2628808135a0" gracePeriod=30
Oct 30 22:29:08.891852 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:08.891817    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"]
Oct 30 22:29:08.892142 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:08.892104    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k" podUID="bd1e5c9d-b6f4-4326-8b97-9f0087c600c7" containerName="istio-proxy" containerID="cri-o://cef8f79b554872c3647b8b56eda2e7064a8bd109319a539243facf040b2c124c" gracePeriod=30
Oct 30 22:29:09.322469 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:09.322416    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"]
Oct 30 22:29:09.333498 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:09.333452    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh" podUID="89d5a590-0ac8-4be8-a791-942b739fc869" containerName="istio-proxy" containerID="cri-o://4a986469abd761f96f6a05beb519ef3ece3e499962f1db141cc3ac17f0a8b90c" gracePeriod=30
Oct 30 22:29:09.360752 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:09.359597    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"]
Oct 30 22:29:09.360752 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:09.359876    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw" podUID="d441e1d3-9571-4513-8404-2d6569566ba9" containerName="istio-proxy" containerID="cri-o://052197b16869f136f5ff20a6390908b88fbaf7f035d1d121f3a066b2a867b073" gracePeriod=30
Oct 30 22:29:09.371367 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:09.371332    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"]
Oct 30 22:29:09.371517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:09.371412    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"]
Oct 30 22:29:09.371680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:09.371630    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2" podUID="3935b91c-508a-4e8d-99f2-23281921afba" containerName="istio-proxy" containerID="cri-o://611eb33d36595091c65fa3a504b77762a888833dd3fb07c2b191e0d2658937f4" gracePeriod=30
Oct 30 22:29:09.371923 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:09.371897    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8" podUID="f08b855f-771b-4002-9555-c7d56ba510c8" containerName="istio-proxy" containerID="cri-o://6813827c1eac5f19ef56ed93458c0c58a68ccd529f1cd2af91606a58b0d53a4f" gracePeriod=30
Oct 30 22:29:09.672297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:09.672197    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:10.169256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.169111    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"]
Oct 30 22:29:10.207256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.206499    4889 cpu_manager.go:414] "containerMap: removing container" podUID="800dedce-28ae-4996-ac41-0253054aea5a" containerName="payload-pre-processing"
Oct 30 22:29:10.207256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.206531    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="800dedce-28ae-4996-ac41-0253054aea5a" containerName="payload-pre-processing"
Oct 30 22:29:10.207256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.206643    4889 cpu_manager.go:414] "containerMap: removing container" podUID="e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29" containerName="payload-processing"
Oct 30 22:29:10.207256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.206653    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29" containerName="payload-processing"
Oct 30 22:29:10.207256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.206902    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8e63a57-6ca5-4ce2-8843-8cf9f41e3c29" containerName="payload-processing"
Oct 30 22:29:10.207256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.206930    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="800dedce-28ae-4996-ac41-0253054aea5a" containerName="payload-pre-processing"
Oct 30 22:29:10.231254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.228124    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"]
Oct 30 22:29:10.231254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.228283    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.258685 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.258658    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-370ec2-openshift-default-dockercfg-lcw59\""
Oct 30 22:29:10.436585 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.431932    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.436585 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.432001    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf853efa-c753-4ce6-834d-c315fba26177-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.436585 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.432072    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf853efa-c753-4ce6-834d-c315fba26177-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.436585 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.432123    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.436585 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.432305    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.436585 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.432420    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf853efa-c753-4ce6-834d-c315fba26177-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.436585 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.432456    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.436585 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.432487    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.436585 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.432560    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwxjq\" (UniqueName: \"kubernetes.io/projected/cf853efa-c753-4ce6-834d-c315fba26177-kube-api-access-nwxjq\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.436585 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.433327    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:10.535250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.534956    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.535250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.535035    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf853efa-c753-4ce6-834d-c315fba26177-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.535250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.535197    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf853efa-c753-4ce6-834d-c315fba26177-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.535560 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.535318    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.550244 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.535918    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.550244 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.536058    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf853efa-c753-4ce6-834d-c315fba26177-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.550244 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.536094    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.550244 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.536124    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.550244 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.539692    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.550244 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.541645    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwxjq\" (UniqueName: \"kubernetes.io/projected/cf853efa-c753-4ce6-834d-c315fba26177-kube-api-access-nwxjq\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.550244 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.542542    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.550244 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.543383    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.550244 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.544522    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf853efa-c753-4ce6-834d-c315fba26177-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.550244 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.544781    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.550244 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.550117    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.616257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.575008    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf853efa-c753-4ce6-834d-c315fba26177-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.682591 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.682552    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwxjq\" (UniqueName: \"kubernetes.io/projected/cf853efa-c753-4ce6-834d-c315fba26177-kube-api-access-nwxjq\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.769250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.769131    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf853efa-c753-4ce6-834d-c315fba26177-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:10.787341 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:29:10.787297    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:29:10.805848 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.805769    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:10.955084 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:10.955043    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:11.002532 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:11.002428    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:11.381523 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:11.381485    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"]
Oct 30 22:29:11.381713 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:29:11.381691    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf853efa_c753_4ce6_834d_c315fba26177.slice/crio-db20aa1315b1b25971caefb69637982b86cc3aede77b66de424387f55af1f25a WatchSource:0}: Error finding container db20aa1315b1b25971caefb69637982b86cc3aede77b66de424387f55af1f25a: Status 404 returned error can't find the container with id db20aa1315b1b25971caefb69637982b86cc3aede77b66de424387f55af1f25a
Oct 30 22:29:11.384879 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:11.384842    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:29:11.385074 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:11.385041    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:29:11.385175 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:11.385082    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:29:11.537693 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:11.537646    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t" event={"ID":"cf853efa-c753-4ce6-834d-c315fba26177","Type":"ContainerStarted","Data":"e6213e7a04157ee84a7d9dc781c8bc7e201caac96e85d397e91060fa41908c57"}
Oct 30 22:29:11.537877 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:11.537725    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t" event={"ID":"cf853efa-c753-4ce6-834d-c315fba26177","Type":"ContainerStarted","Data":"db20aa1315b1b25971caefb69637982b86cc3aede77b66de424387f55af1f25a"}
Oct 30 22:29:11.579940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:11.579885    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t" podStartSLOduration=1.579869373 podStartE2EDuration="1.579869373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:11.57746606 +0000 UTC m=+1336.933832373" watchObservedRunningTime="2026-10-30 22:29:11.579869373 +0000 UTC m=+1336.936235686"
Oct 30 22:29:11.956137 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:11.956108    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:12.439731 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:12.439694    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:12.962115 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:12.962086    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:13.564330 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:13.564292    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:13.565359 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:13.565340    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:29:14.066311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.066189    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:14.235207 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.235026    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:29:14.241151 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.241089    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:14.338492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.329027    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f63c047-c985-45ed-92b5-3fed61c84385-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f63c047-c985-45ed-92b5-3fed61c84385-istiod-ca-cert\") pod \"4f63c047-c985-45ed-92b5-3fed61c84385\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") "
Oct 30 22:29:14.338492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.329103    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-credential-socket\") pod \"4f63c047-c985-45ed-92b5-3fed61c84385\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") "
Oct 30 22:29:14.338492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.329197    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f63c047-c985-45ed-92b5-3fed61c84385-istio-token\" (UniqueName: \"kubernetes.io/projected/4f63c047-c985-45ed-92b5-3fed61c84385-istio-token\") pod \"4f63c047-c985-45ed-92b5-3fed61c84385\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") "
Oct 30 22:29:14.338492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.329360    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-istio-data\") pod \"4f63c047-c985-45ed-92b5-3fed61c84385\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") "
Oct 30 22:29:14.338492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.329436    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-workload-socket\") pod \"4f63c047-c985-45ed-92b5-3fed61c84385\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") "
Oct 30 22:29:14.338492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.330100    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f63c047-c985-45ed-92b5-3fed61c84385-istiod-ca-cert" pod "4f63c047-c985-45ed-92b5-3fed61c84385" (UID: "4f63c047-c985-45ed-92b5-3fed61c84385"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:14.338492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.330695    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-credential-socket" pod "4f63c047-c985-45ed-92b5-3fed61c84385" (UID: "4f63c047-c985-45ed-92b5-3fed61c84385"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.338492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.330912    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-istio-data" pod "4f63c047-c985-45ed-92b5-3fed61c84385" (UID: "4f63c047-c985-45ed-92b5-3fed61c84385"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.338492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.331141    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-workload-socket" pod "4f63c047-c985-45ed-92b5-3fed61c84385" (UID: "4f63c047-c985-45ed-92b5-3fed61c84385"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.338492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.331543    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-workload-certs\") pod \"4f63c047-c985-45ed-92b5-3fed61c84385\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") "
Oct 30 22:29:14.338492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.331626    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-istio-envoy\") pod \"4f63c047-c985-45ed-92b5-3fed61c84385\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") "
Oct 30 22:29:14.338492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.331678    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f63c047-c985-45ed-92b5-3fed61c84385-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f63c047-c985-45ed-92b5-3fed61c84385-istio-podinfo\") pod \"4f63c047-c985-45ed-92b5-3fed61c84385\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") "
Oct 30 22:29:14.338492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.331703    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f63c047-c985-45ed-92b5-3fed61c84385-kube-api-access-jfzxr\" (UniqueName: \"kubernetes.io/projected/4f63c047-c985-45ed-92b5-3fed61c84385-kube-api-access-jfzxr\") pod \"4f63c047-c985-45ed-92b5-3fed61c84385\" (UID: \"4f63c047-c985-45ed-92b5-3fed61c84385\") "
Oct 30 22:29:14.338492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.332660    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f63c047-c985-45ed-92b5-3fed61c84385-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.338492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.332684    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.338492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.332700    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.338492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.332714    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.347542 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.347474    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-workload-certs" pod "4f63c047-c985-45ed-92b5-3fed61c84385" (UID: "4f63c047-c985-45ed-92b5-3fed61c84385"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.423191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.422777    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:14.437521 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.435435    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.499506 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.499429    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-istio-envoy" pod "4f63c047-c985-45ed-92b5-3fed61c84385" (UID: "4f63c047-c985-45ed-92b5-3fed61c84385"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.499506 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.499435    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f63c047-c985-45ed-92b5-3fed61c84385-istio-token" pod "4f63c047-c985-45ed-92b5-3fed61c84385" (UID: "4f63c047-c985-45ed-92b5-3fed61c84385"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:14.506617 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.506575    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f63c047-c985-45ed-92b5-3fed61c84385-istio-podinfo" pod "4f63c047-c985-45ed-92b5-3fed61c84385" (UID: "4f63c047-c985-45ed-92b5-3fed61c84385"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:14.518093 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.518066    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f63c047-c985-45ed-92b5-3fed61c84385-kube-api-access-jfzxr" pod "4f63c047-c985-45ed-92b5-3fed61c84385" (UID: "4f63c047-c985-45ed-92b5-3fed61c84385"). InnerVolumeSpecName "kube-api-access-jfzxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:14.541270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.541171    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f63c047-c985-45ed-92b5-3fed61c84385-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.541270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.541200    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f63c047-c985-45ed-92b5-3fed61c84385-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.541270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.541214    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f63c047-c985-45ed-92b5-3fed61c84385-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.541270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.541245    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfzxr\" (UniqueName: \"kubernetes.io/projected/4f63c047-c985-45ed-92b5-3fed61c84385-kube-api-access-jfzxr\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.578410 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.578113    4889 generic.go:356] "Generic (PLEG): container finished" podID="4f63c047-c985-45ed-92b5-3fed61c84385" containerID="967d05a6cea8cfdee6de2d406efa400eae39299234f916036c8e6f633b13d631" exitCode=0
Oct 30 22:29:14.578410 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.578241    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k" event={"ID":"4f63c047-c985-45ed-92b5-3fed61c84385","Type":"ContainerDied","Data":"967d05a6cea8cfdee6de2d406efa400eae39299234f916036c8e6f633b13d631"}
Oct 30 22:29:14.578410 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.578296    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k" event={"ID":"4f63c047-c985-45ed-92b5-3fed61c84385","Type":"ContainerDied","Data":"ee5321f714e07cd83e8a6a38c4613efa3fd6476021925986a8a68ec911352e5a"}
Oct 30 22:29:14.578410 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.578328    4889 scope.go:122] "RemoveContainer" containerID="967d05a6cea8cfdee6de2d406efa400eae39299234f916036c8e6f633b13d631"
Oct 30 22:29:14.580702 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.580307    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"
Oct 30 22:29:14.593770 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.593187    4889 generic.go:356] "Generic (PLEG): container finished" podID="89d5a590-0ac8-4be8-a791-942b739fc869" containerID="4a986469abd761f96f6a05beb519ef3ece3e499962f1db141cc3ac17f0a8b90c" exitCode=0
Oct 30 22:29:14.593770 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.593311    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh" event={"ID":"89d5a590-0ac8-4be8-a791-942b739fc869","Type":"ContainerDied","Data":"4a986469abd761f96f6a05beb519ef3ece3e499962f1db141cc3ac17f0a8b90c"}
Oct 30 22:29:14.596695 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.596510    4889 generic.go:356] "Generic (PLEG): container finished" podID="3935b91c-508a-4e8d-99f2-23281921afba" containerID="611eb33d36595091c65fa3a504b77762a888833dd3fb07c2b191e0d2658937f4" exitCode=0
Oct 30 22:29:14.596695 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.596607    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2" event={"ID":"3935b91c-508a-4e8d-99f2-23281921afba","Type":"ContainerDied","Data":"611eb33d36595091c65fa3a504b77762a888833dd3fb07c2b191e0d2658937f4"}
Oct 30 22:29:14.602485 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.602147    4889 generic.go:356] "Generic (PLEG): container finished" podID="d441e1d3-9571-4513-8404-2d6569566ba9" containerID="052197b16869f136f5ff20a6390908b88fbaf7f035d1d121f3a066b2a867b073" exitCode=0
Oct 30 22:29:14.602485 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.602415    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw" event={"ID":"d441e1d3-9571-4513-8404-2d6569566ba9","Type":"ContainerDied","Data":"052197b16869f136f5ff20a6390908b88fbaf7f035d1d121f3a066b2a867b073"}
Oct 30 22:29:14.604467 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.604255    4889 generic.go:356] "Generic (PLEG): container finished" podID="408d956d-ee1c-44e8-9bc5-8d3fc72ab890" containerID="d42888dc1dab5f808a283a03d24fc1c473662a6fcecf7d7ccf3d2628808135a0" exitCode=0
Oct 30 22:29:14.604467 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.604345    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv" event={"ID":"408d956d-ee1c-44e8-9bc5-8d3fc72ab890","Type":"ContainerDied","Data":"d42888dc1dab5f808a283a03d24fc1c473662a6fcecf7d7ccf3d2628808135a0"}
Oct 30 22:29:14.604467 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.604386    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv" event={"ID":"408d956d-ee1c-44e8-9bc5-8d3fc72ab890","Type":"ContainerDied","Data":"89caee667c866bb1e6b967622514230eff9143cd61b3c41bb16d8f4d8ea878ff"}
Oct 30 22:29:14.604467 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.604409    4889 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89caee667c866bb1e6b967622514230eff9143cd61b3c41bb16d8f4d8ea878ff"
Oct 30 22:29:14.608572 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.606587    4889 generic.go:356] "Generic (PLEG): container finished" podID="08eb9eea-09b6-4149-9c58-5a252ef3062a" containerID="802fd5d0b6f4a8a2dd4205d5e826ce855238318dc56569db2cfac5f641f8d94d" exitCode=0
Oct 30 22:29:14.608572 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.606663    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8" event={"ID":"08eb9eea-09b6-4149-9c58-5a252ef3062a","Type":"ContainerDied","Data":"802fd5d0b6f4a8a2dd4205d5e826ce855238318dc56569db2cfac5f641f8d94d"}
Oct 30 22:29:14.608572 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.606688    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8" event={"ID":"08eb9eea-09b6-4149-9c58-5a252ef3062a","Type":"ContainerDied","Data":"bb103cc1c89b7d16adbe28098777ba8cd40ac23f366093ab1e79b08a3cc5ff4c"}
Oct 30 22:29:14.608572 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.606702    4889 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb103cc1c89b7d16adbe28098777ba8cd40ac23f366093ab1e79b08a3cc5ff4c"
Oct 30 22:29:14.609419 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.608949    4889 generic.go:356] "Generic (PLEG): container finished" podID="f08b855f-771b-4002-9555-c7d56ba510c8" containerID="6813827c1eac5f19ef56ed93458c0c58a68ccd529f1cd2af91606a58b0d53a4f" exitCode=0
Oct 30 22:29:14.609419 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.609020    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8" event={"ID":"f08b855f-771b-4002-9555-c7d56ba510c8","Type":"ContainerDied","Data":"6813827c1eac5f19ef56ed93458c0c58a68ccd529f1cd2af91606a58b0d53a4f"}
Oct 30 22:29:14.611986 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.611056    4889 generic.go:356] "Generic (PLEG): container finished" podID="bd1e5c9d-b6f4-4326-8b97-9f0087c600c7" containerID="cef8f79b554872c3647b8b56eda2e7064a8bd109319a539243facf040b2c124c" exitCode=0
Oct 30 22:29:14.611986 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.611310    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k" event={"ID":"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7","Type":"ContainerDied","Data":"cef8f79b554872c3647b8b56eda2e7064a8bd109319a539243facf040b2c124c"}
Oct 30 22:29:14.611986 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.611352    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k" event={"ID":"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7","Type":"ContainerDied","Data":"092efeb1f1d53fbbaaed80f82b05242164ed59b80fa8db85d8b8a189d731a7e5"}
Oct 30 22:29:14.611986 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.611374    4889 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="092efeb1f1d53fbbaaed80f82b05242164ed59b80fa8db85d8b8a189d731a7e5"
Oct 30 22:29:14.653924 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.653903    4889 scope.go:122] "RemoveContainer" containerID="967d05a6cea8cfdee6de2d406efa400eae39299234f916036c8e6f633b13d631"
Oct 30 22:29:14.654773 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:29:14.654704    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"967d05a6cea8cfdee6de2d406efa400eae39299234f916036c8e6f633b13d631\": container with ID starting with 967d05a6cea8cfdee6de2d406efa400eae39299234f916036c8e6f633b13d631 not found: ID does not exist" containerID="967d05a6cea8cfdee6de2d406efa400eae39299234f916036c8e6f633b13d631"
Oct 30 22:29:14.654773 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.654743    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967d05a6cea8cfdee6de2d406efa400eae39299234f916036c8e6f633b13d631"} err="failed to get container status \"967d05a6cea8cfdee6de2d406efa400eae39299234f916036c8e6f633b13d631\": rpc error: code = NotFound desc = could not find container \"967d05a6cea8cfdee6de2d406efa400eae39299234f916036c8e6f633b13d631\": container with ID starting with 967d05a6cea8cfdee6de2d406efa400eae39299234f916036c8e6f633b13d631 not found: ID does not exist"
Oct 30 22:29:14.672263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.672240    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:29:14.675140 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.674398    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:29:14.677405 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.677340    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:29:14.705655 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.705581    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"]
Oct 30 22:29:14.712312 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.712282    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-bhn2k"]
Oct 30 22:29:14.745803 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.745770    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-token\" (UniqueName: \"kubernetes.io/projected/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-token\") pod \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") "
Oct 30 22:29:14.745980 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.745826    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istiod-ca-cert\") pod \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") "
Oct 30 22:29:14.745980 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.745879    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-token\" (UniqueName: \"kubernetes.io/projected/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-token\") pod \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") "
Oct 30 22:29:14.745980 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.745914    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-envoy\") pod \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") "
Oct 30 22:29:14.746148 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.745996    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-workload-certs\") pod \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") "
Oct 30 22:29:14.746148 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.746056    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istiod-ca-cert\") pod \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") "
Oct 30 22:29:14.746148 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.746086    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-workload-certs\") pod \"08eb9eea-09b6-4149-9c58-5a252ef3062a\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") "
Oct 30 22:29:14.746148 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.746140    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-credential-socket\") pod \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") "
Oct 30 22:29:14.746374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.746171    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-envoy\") pod \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") "
Oct 30 22:29:14.746374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.746211    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-podinfo\") pod \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") "
Oct 30 22:29:14.746374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.746304    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-data\") pod \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") "
Oct 30 22:29:14.746374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.746334    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08eb9eea-09b6-4149-9c58-5a252ef3062a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08eb9eea-09b6-4149-9c58-5a252ef3062a-istiod-ca-cert\") pod \"08eb9eea-09b6-4149-9c58-5a252ef3062a\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") "
Oct 30 22:29:14.746578 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.746377    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08eb9eea-09b6-4149-9c58-5a252ef3062a-kube-api-access-22gwf\" (UniqueName: \"kubernetes.io/projected/08eb9eea-09b6-4149-9c58-5a252ef3062a-kube-api-access-22gwf\") pod \"08eb9eea-09b6-4149-9c58-5a252ef3062a\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") "
Oct 30 22:29:14.746578 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.746418    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-podinfo\") pod \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") "
Oct 30 22:29:14.746578 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.746490    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-data\") pod \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") "
Oct 30 22:29:14.746578 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.746528    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-podinfo\") pod \"08eb9eea-09b6-4149-9c58-5a252ef3062a\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") "
Oct 30 22:29:14.746779 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.746632    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-workload-socket\") pod \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") "
Oct 30 22:29:14.746779 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.746661    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-workload-socket\") pod \"08eb9eea-09b6-4149-9c58-5a252ef3062a\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") "
Oct 30 22:29:14.746779 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.746725    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-kube-api-access-mvf6p\" (UniqueName: \"kubernetes.io/projected/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-kube-api-access-mvf6p\") pod \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") "
Oct 30 22:29:14.746927 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.746776    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-workload-socket\") pod \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") "
Oct 30 22:29:14.746927 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.746817    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-token\" (UniqueName: \"kubernetes.io/projected/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-token\") pod \"08eb9eea-09b6-4149-9c58-5a252ef3062a\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") "
Oct 30 22:29:14.746927 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.746882    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-credential-socket\") pod \"08eb9eea-09b6-4149-9c58-5a252ef3062a\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") "
Oct 30 22:29:14.747082 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.746927    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-kube-api-access-rqbx6\" (UniqueName: \"kubernetes.io/projected/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-kube-api-access-rqbx6\") pod \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") "
Oct 30 22:29:14.747082 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.747061    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-credential-socket\") pod \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\" (UID: \"408d956d-ee1c-44e8-9bc5-8d3fc72ab890\") "
Oct 30 22:29:14.747185 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.747092    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-workload-certs\") pod \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\" (UID: \"bd1e5c9d-b6f4-4326-8b97-9f0087c600c7\") "
Oct 30 22:29:14.747294 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.747189    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-data\") pod \"08eb9eea-09b6-4149-9c58-5a252ef3062a\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") "
Oct 30 22:29:14.747294 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.747248    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-envoy\") pod \"08eb9eea-09b6-4149-9c58-5a252ef3062a\" (UID: \"08eb9eea-09b6-4149-9c58-5a252ef3062a\") "
Oct 30 22:29:14.749024 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.748364    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-workload-certs" pod "08eb9eea-09b6-4149-9c58-5a252ef3062a" (UID: "08eb9eea-09b6-4149-9c58-5a252ef3062a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.749024 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.748650    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-credential-socket" pod "08eb9eea-09b6-4149-9c58-5a252ef3062a" (UID: "08eb9eea-09b6-4149-9c58-5a252ef3062a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.749024 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.748728    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istiod-ca-cert" pod "408d956d-ee1c-44e8-9bc5-8d3fc72ab890" (UID: "408d956d-ee1c-44e8-9bc5-8d3fc72ab890"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:14.749213 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.749098    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-data" pod "408d956d-ee1c-44e8-9bc5-8d3fc72ab890" (UID: "408d956d-ee1c-44e8-9bc5-8d3fc72ab890"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.749588 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.749564    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-credential-socket" pod "bd1e5c9d-b6f4-4326-8b97-9f0087c600c7" (UID: "bd1e5c9d-b6f4-4326-8b97-9f0087c600c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.751399 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.750598    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-podinfo" pod "408d956d-ee1c-44e8-9bc5-8d3fc72ab890" (UID: "408d956d-ee1c-44e8-9bc5-8d3fc72ab890"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:14.751399 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.750814    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-data" pod "bd1e5c9d-b6f4-4326-8b97-9f0087c600c7" (UID: "bd1e5c9d-b6f4-4326-8b97-9f0087c600c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.751399 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.751111    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08eb9eea-09b6-4149-9c58-5a252ef3062a-istiod-ca-cert" pod "08eb9eea-09b6-4149-9c58-5a252ef3062a" (UID: "08eb9eea-09b6-4149-9c58-5a252ef3062a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:14.752349 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.751708    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:29:14.752349 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.751998    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-workload-socket" pod "bd1e5c9d-b6f4-4326-8b97-9f0087c600c7" (UID: "bd1e5c9d-b6f4-4326-8b97-9f0087c600c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.752349 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.752140    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-workload-certs" pod "408d956d-ee1c-44e8-9bc5-8d3fc72ab890" (UID: "408d956d-ee1c-44e8-9bc5-8d3fc72ab890"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.752998 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.752749    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-workload-socket" pod "408d956d-ee1c-44e8-9bc5-8d3fc72ab890" (UID: "408d956d-ee1c-44e8-9bc5-8d3fc72ab890"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.754367 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.754100    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-workload-socket" pod "08eb9eea-09b6-4149-9c58-5a252ef3062a" (UID: "08eb9eea-09b6-4149-9c58-5a252ef3062a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.756617 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.755412    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-podinfo" pod "08eb9eea-09b6-4149-9c58-5a252ef3062a" (UID: "08eb9eea-09b6-4149-9c58-5a252ef3062a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:14.756617 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.755214    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08eb9eea-09b6-4149-9c58-5a252ef3062a-kube-api-access-22gwf" pod "08eb9eea-09b6-4149-9c58-5a252ef3062a" (UID: "08eb9eea-09b6-4149-9c58-5a252ef3062a"). InnerVolumeSpecName "kube-api-access-22gwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:14.756617 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.755952    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istiod-ca-cert" pod "bd1e5c9d-b6f4-4326-8b97-9f0087c600c7" (UID: "bd1e5c9d-b6f4-4326-8b97-9f0087c600c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:14.756617 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.756138    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-token" pod "08eb9eea-09b6-4149-9c58-5a252ef3062a" (UID: "08eb9eea-09b6-4149-9c58-5a252ef3062a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:14.756617 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.756405    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-workload-certs" pod "bd1e5c9d-b6f4-4326-8b97-9f0087c600c7" (UID: "bd1e5c9d-b6f4-4326-8b97-9f0087c600c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.756617 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.756497    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-credential-socket" pod "408d956d-ee1c-44e8-9bc5-8d3fc72ab890" (UID: "408d956d-ee1c-44e8-9bc5-8d3fc72ab890"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.758362 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.756624    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-data" pod "08eb9eea-09b6-4149-9c58-5a252ef3062a" (UID: "08eb9eea-09b6-4149-9c58-5a252ef3062a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.758362 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.757961    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-token" pod "408d956d-ee1c-44e8-9bc5-8d3fc72ab890" (UID: "408d956d-ee1c-44e8-9bc5-8d3fc72ab890"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:14.758362 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.758060    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-token" pod "bd1e5c9d-b6f4-4326-8b97-9f0087c600c7" (UID: "bd1e5c9d-b6f4-4326-8b97-9f0087c600c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:14.758362 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.758110    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-podinfo" pod "bd1e5c9d-b6f4-4326-8b97-9f0087c600c7" (UID: "bd1e5c9d-b6f4-4326-8b97-9f0087c600c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:14.758362 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.758148    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-kube-api-access-rqbx6" pod "bd1e5c9d-b6f4-4326-8b97-9f0087c600c7" (UID: "bd1e5c9d-b6f4-4326-8b97-9f0087c600c7"). InnerVolumeSpecName "kube-api-access-rqbx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:14.758362 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.758211    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-kube-api-access-mvf6p" pod "408d956d-ee1c-44e8-9bc5-8d3fc72ab890" (UID: "408d956d-ee1c-44e8-9bc5-8d3fc72ab890"). InnerVolumeSpecName "kube-api-access-mvf6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:14.758875 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.758850    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-envoy" pod "08eb9eea-09b6-4149-9c58-5a252ef3062a" (UID: "08eb9eea-09b6-4149-9c58-5a252ef3062a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.759753 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.759724    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-envoy" pod "408d956d-ee1c-44e8-9bc5-8d3fc72ab890" (UID: "408d956d-ee1c-44e8-9bc5-8d3fc72ab890"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.762362 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.762324    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-envoy" pod "bd1e5c9d-b6f4-4326-8b97-9f0087c600c7" (UID: "bd1e5c9d-b6f4-4326-8b97-9f0087c600c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.788479 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.787970    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f63c047-c985-45ed-92b5-3fed61c84385" path="/var/lib/kubelet/pods/4f63c047-c985-45ed-92b5-3fed61c84385/volumes"
Oct 30 22:29:14.794815 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.794050    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:29:14.851282 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.851187    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89d5a590-0ac8-4be8-a791-942b739fc869-kube-api-access-p48px\" (UniqueName: \"kubernetes.io/projected/89d5a590-0ac8-4be8-a791-942b739fc869-kube-api-access-p48px\") pod \"89d5a590-0ac8-4be8-a791-942b739fc869\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") "
Oct 30 22:29:14.851557 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.851537    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-istio-data\") pod \"3935b91c-508a-4e8d-99f2-23281921afba\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") "
Oct 30 22:29:14.851842 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.851817    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3935b91c-508a-4e8d-99f2-23281921afba-kube-api-access-tnw2h\" (UniqueName: \"kubernetes.io/projected/3935b91c-508a-4e8d-99f2-23281921afba-kube-api-access-tnw2h\") pod \"3935b91c-508a-4e8d-99f2-23281921afba\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") "
Oct 30 22:29:14.852068 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.852011    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3935b91c-508a-4e8d-99f2-23281921afba-istio-token\" (UniqueName: \"kubernetes.io/projected/3935b91c-508a-4e8d-99f2-23281921afba-istio-token\") pod \"3935b91c-508a-4e8d-99f2-23281921afba\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") "
Oct 30 22:29:14.852284 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.852267    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3935b91c-508a-4e8d-99f2-23281921afba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3935b91c-508a-4e8d-99f2-23281921afba-istio-podinfo\") pod \"3935b91c-508a-4e8d-99f2-23281921afba\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") "
Oct 30 22:29:14.852395 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.852382    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3935b91c-508a-4e8d-99f2-23281921afba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3935b91c-508a-4e8d-99f2-23281921afba-istiod-ca-cert\") pod \"3935b91c-508a-4e8d-99f2-23281921afba\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") "
Oct 30 22:29:14.852507 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.852495    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89d5a590-0ac8-4be8-a791-942b739fc869-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d5a590-0ac8-4be8-a791-942b739fc869-istio-podinfo\") pod \"89d5a590-0ac8-4be8-a791-942b739fc869\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") "
Oct 30 22:29:14.852664 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.852652    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-workload-certs\") pod \"89d5a590-0ac8-4be8-a791-942b739fc869\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") "
Oct 30 22:29:14.852779 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.852766    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-workload-certs\") pod \"3935b91c-508a-4e8d-99f2-23281921afba\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") "
Oct 30 22:29:14.852891 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.852880    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-workload-socket\") pod \"89d5a590-0ac8-4be8-a791-942b739fc869\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") "
Oct 30 22:29:14.853009 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.852998    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-workload-socket\") pod \"3935b91c-508a-4e8d-99f2-23281921afba\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") "
Oct 30 22:29:14.864053 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.862950    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89d5a590-0ac8-4be8-a791-942b739fc869-istio-token\" (UniqueName: \"kubernetes.io/projected/89d5a590-0ac8-4be8-a791-942b739fc869-istio-token\") pod \"89d5a590-0ac8-4be8-a791-942b739fc869\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") "
Oct 30 22:29:14.868386 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.868349    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-credential-socket\") pod \"3935b91c-508a-4e8d-99f2-23281921afba\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") "
Oct 30 22:29:14.868582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.868567    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-istio-envoy\") pod \"3935b91c-508a-4e8d-99f2-23281921afba\" (UID: \"3935b91c-508a-4e8d-99f2-23281921afba\") "
Oct 30 22:29:14.869042 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.859754    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3935b91c-508a-4e8d-99f2-23281921afba-istio-podinfo" pod "3935b91c-508a-4e8d-99f2-23281921afba" (UID: "3935b91c-508a-4e8d-99f2-23281921afba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:14.869042 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.861578    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-workload-certs" pod "89d5a590-0ac8-4be8-a791-942b739fc869" (UID: "89d5a590-0ac8-4be8-a791-942b739fc869"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.869042 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.861897    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3935b91c-508a-4e8d-99f2-23281921afba-istiod-ca-cert" pod "3935b91c-508a-4e8d-99f2-23281921afba" (UID: "3935b91c-508a-4e8d-99f2-23281921afba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:14.869042 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.863098    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d5a590-0ac8-4be8-a791-942b739fc869-kube-api-access-p48px" pod "89d5a590-0ac8-4be8-a791-942b739fc869" (UID: "89d5a590-0ac8-4be8-a791-942b739fc869"). InnerVolumeSpecName "kube-api-access-p48px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:14.869042 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.863310    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-istio-data" pod "3935b91c-508a-4e8d-99f2-23281921afba" (UID: "3935b91c-508a-4e8d-99f2-23281921afba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.869042 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.863636    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-workload-socket" pod "89d5a590-0ac8-4be8-a791-942b739fc869" (UID: "89d5a590-0ac8-4be8-a791-942b739fc869"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.869042 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.863803    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-workload-certs" pod "3935b91c-508a-4e8d-99f2-23281921afba" (UID: "3935b91c-508a-4e8d-99f2-23281921afba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.869042 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.866813    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-workload-socket" pod "3935b91c-508a-4e8d-99f2-23281921afba" (UID: "3935b91c-508a-4e8d-99f2-23281921afba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.869042 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.867350    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89d5a590-0ac8-4be8-a791-942b739fc869-istio-podinfo" pod "89d5a590-0ac8-4be8-a791-942b739fc869" (UID: "89d5a590-0ac8-4be8-a791-942b739fc869"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:14.869042 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.868675    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-credential-socket" pod "3935b91c-508a-4e8d-99f2-23281921afba" (UID: "3935b91c-508a-4e8d-99f2-23281921afba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.869042 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.868964    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89d5a590-0ac8-4be8-a791-942b739fc869-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d5a590-0ac8-4be8-a791-942b739fc869-istiod-ca-cert\") pod \"89d5a590-0ac8-4be8-a791-942b739fc869\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") "
Oct 30 22:29:14.869042 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.868997    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-istio-data\") pod \"89d5a590-0ac8-4be8-a791-942b739fc869\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") "
Oct 30 22:29:14.870367 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.869529    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-istio-envoy\") pod \"89d5a590-0ac8-4be8-a791-942b739fc869\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") "
Oct 30 22:29:14.872401 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.870164    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-credential-socket\") pod \"89d5a590-0ac8-4be8-a791-942b739fc869\" (UID: \"89d5a590-0ac8-4be8-a791-942b739fc869\") "
Oct 30 22:29:14.874722 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.869930    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-istio-data" pod "89d5a590-0ac8-4be8-a791-942b739fc869" (UID: "89d5a590-0ac8-4be8-a791-942b739fc869"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.874722 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.870551    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-credential-socket" pod "89d5a590-0ac8-4be8-a791-942b739fc869" (UID: "89d5a590-0ac8-4be8-a791-942b739fc869"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.874722 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.872354    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89d5a590-0ac8-4be8-a791-942b739fc869-istiod-ca-cert" pod "89d5a590-0ac8-4be8-a791-942b739fc869" (UID: "89d5a590-0ac8-4be8-a791-942b739fc869"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:14.883582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.883543    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897387 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897360    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897387 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897389    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897560 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897402    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3935b91c-508a-4e8d-99f2-23281921afba-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897560 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897414    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3935b91c-508a-4e8d-99f2-23281921afba-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897560 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.886108    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d5a590-0ac8-4be8-a791-942b739fc869-istio-token" pod "89d5a590-0ac8-4be8-a791-942b739fc869" (UID: "89d5a590-0ac8-4be8-a791-942b739fc869"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:14.897560 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897451    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897560 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897465    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897560 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897478    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d5a590-0ac8-4be8-a791-942b739fc869-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897560 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897490    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897560 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897502    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897560 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897516    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08eb9eea-09b6-4149-9c58-5a252ef3062a-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897560 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897529    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22gwf\" (UniqueName: \"kubernetes.io/projected/08eb9eea-09b6-4149-9c58-5a252ef3062a-kube-api-access-22gwf\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897560 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897541    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897560 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897555    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897568    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897583    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897597    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897610    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897622    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897636    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897650    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897664    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897677    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d5a590-0ac8-4be8-a791-942b739fc869-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897689    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897705    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvf6p\" (UniqueName: \"kubernetes.io/projected/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-kube-api-access-mvf6p\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897719    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897732    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897745    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897758    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqbx6\" (UniqueName: \"kubernetes.io/projected/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-kube-api-access-rqbx6\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897773    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897787    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897800    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897814    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p48px\" (UniqueName: \"kubernetes.io/projected/89d5a590-0ac8-4be8-a791-942b739fc869-kube-api-access-p48px\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897827    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897839    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08eb9eea-09b6-4149-9c58-5a252ef3062a-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897851    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897863    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897874    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897886    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.897970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.897898    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408d956d-ee1c-44e8-9bc5-8d3fc72ab890-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.908468 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.908390    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-istio-envoy" pod "3935b91c-508a-4e8d-99f2-23281921afba" (UID: "3935b91c-508a-4e8d-99f2-23281921afba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.908468 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.908408    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3935b91c-508a-4e8d-99f2-23281921afba-kube-api-access-tnw2h" pod "3935b91c-508a-4e8d-99f2-23281921afba" (UID: "3935b91c-508a-4e8d-99f2-23281921afba"). InnerVolumeSpecName "kube-api-access-tnw2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:14.908468 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.908432    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-istio-envoy" pod "89d5a590-0ac8-4be8-a791-942b739fc869" (UID: "89d5a590-0ac8-4be8-a791-942b739fc869"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.908468 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:14.908439    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3935b91c-508a-4e8d-99f2-23281921afba-istio-token" pod "3935b91c-508a-4e8d-99f2-23281921afba" (UID: "3935b91c-508a-4e8d-99f2-23281921afba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:15.013486 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.013457    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89d5a590-0ac8-4be8-a791-942b739fc869-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.013733 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.013718    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3935b91c-508a-4e8d-99f2-23281921afba-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.013873 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.013862    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d5a590-0ac8-4be8-a791-942b739fc869-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.014017 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.014005    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnw2h\" (UniqueName: \"kubernetes.io/projected/3935b91c-508a-4e8d-99f2-23281921afba-kube-api-access-tnw2h\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.014149 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.014122    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3935b91c-508a-4e8d-99f2-23281921afba-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.034834 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.034807    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:29:15.036888 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.036865    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:29:15.114921 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.114893    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d441e1d3-9571-4513-8404-2d6569566ba9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d441e1d3-9571-4513-8404-2d6569566ba9-istio-podinfo\") pod \"d441e1d3-9571-4513-8404-2d6569566ba9\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") "
Oct 30 22:29:15.115415 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.115388    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-workload-certs\") pod \"f08b855f-771b-4002-9555-c7d56ba510c8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") "
Oct 30 22:29:15.115540 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.115444    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-workload-socket\") pod \"d441e1d3-9571-4513-8404-2d6569566ba9\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") "
Oct 30 22:29:15.115540 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.115508    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-workload-certs\") pod \"d441e1d3-9571-4513-8404-2d6569566ba9\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") "
Oct 30 22:29:15.115645 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.115541    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-credential-socket\") pod \"d441e1d3-9571-4513-8404-2d6569566ba9\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") "
Oct 30 22:29:15.115645 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.115583    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f08b855f-771b-4002-9555-c7d56ba510c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f08b855f-771b-4002-9555-c7d56ba510c8-istio-podinfo\") pod \"f08b855f-771b-4002-9555-c7d56ba510c8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") "
Oct 30 22:29:15.115645 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.115622    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d441e1d3-9571-4513-8404-2d6569566ba9-kube-api-access-8jmxg\" (UniqueName: \"kubernetes.io/projected/d441e1d3-9571-4513-8404-2d6569566ba9-kube-api-access-8jmxg\") pod \"d441e1d3-9571-4513-8404-2d6569566ba9\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") "
Oct 30 22:29:15.115779 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.115660    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-istio-data\") pod \"f08b855f-771b-4002-9555-c7d56ba510c8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") "
Oct 30 22:29:15.115779 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.115688    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-workload-certs" pod "f08b855f-771b-4002-9555-c7d56ba510c8" (UID: "f08b855f-771b-4002-9555-c7d56ba510c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:15.115779 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.115745    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d441e1d3-9571-4513-8404-2d6569566ba9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d441e1d3-9571-4513-8404-2d6569566ba9-istiod-ca-cert\") pod \"d441e1d3-9571-4513-8404-2d6569566ba9\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") "
Oct 30 22:29:15.115779 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.115760    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d441e1d3-9571-4513-8404-2d6569566ba9-istio-token\" (UniqueName: \"kubernetes.io/projected/d441e1d3-9571-4513-8404-2d6569566ba9-istio-token\") pod \"d441e1d3-9571-4513-8404-2d6569566ba9\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") "
Oct 30 22:29:15.115963 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.115855    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-istio-envoy\") pod \"d441e1d3-9571-4513-8404-2d6569566ba9\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") "
Oct 30 22:29:15.115963 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.115884    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-credential-socket" pod "d441e1d3-9571-4513-8404-2d6569566ba9" (UID: "d441e1d3-9571-4513-8404-2d6569566ba9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:15.115963 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.115907    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f08b855f-771b-4002-9555-c7d56ba510c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f08b855f-771b-4002-9555-c7d56ba510c8-istiod-ca-cert\") pod \"f08b855f-771b-4002-9555-c7d56ba510c8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") "
Oct 30 22:29:15.116106 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.115970    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f08b855f-771b-4002-9555-c7d56ba510c8-kube-api-access-nrtqd\" (UniqueName: \"kubernetes.io/projected/f08b855f-771b-4002-9555-c7d56ba510c8-kube-api-access-nrtqd\") pod \"f08b855f-771b-4002-9555-c7d56ba510c8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") "
Oct 30 22:29:15.116106 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.116003    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-credential-socket\") pod \"f08b855f-771b-4002-9555-c7d56ba510c8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") "
Oct 30 22:29:15.116106 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.116043    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-istio-envoy\") pod \"f08b855f-771b-4002-9555-c7d56ba510c8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") "
Oct 30 22:29:15.116300 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.116118    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-istio-data\") pod \"d441e1d3-9571-4513-8404-2d6569566ba9\" (UID: \"d441e1d3-9571-4513-8404-2d6569566ba9\") "
Oct 30 22:29:15.116300 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.116141    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-workload-socket" pod "d441e1d3-9571-4513-8404-2d6569566ba9" (UID: "d441e1d3-9571-4513-8404-2d6569566ba9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:15.116300 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.116159    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f08b855f-771b-4002-9555-c7d56ba510c8-istio-token\" (UniqueName: \"kubernetes.io/projected/f08b855f-771b-4002-9555-c7d56ba510c8-istio-token\") pod \"f08b855f-771b-4002-9555-c7d56ba510c8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") "
Oct 30 22:29:15.116300 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.116181    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-workload-socket\") pod \"f08b855f-771b-4002-9555-c7d56ba510c8\" (UID: \"f08b855f-771b-4002-9555-c7d56ba510c8\") "
Oct 30 22:29:15.116509 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.116457    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-workload-certs" pod "d441e1d3-9571-4513-8404-2d6569566ba9" (UID: "d441e1d3-9571-4513-8404-2d6569566ba9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:15.116661 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.116632    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-credential-socket" pod "f08b855f-771b-4002-9555-c7d56ba510c8" (UID: "f08b855f-771b-4002-9555-c7d56ba510c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:15.117039 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.117015    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f08b855f-771b-4002-9555-c7d56ba510c8-istiod-ca-cert" pod "f08b855f-771b-4002-9555-c7d56ba510c8" (UID: "f08b855f-771b-4002-9555-c7d56ba510c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:15.117128 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.117023    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.117128 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.117068    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.117128 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.117088    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.117128 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.117103    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.117128 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.117115    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.117643 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.117614    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-istio-data" pod "f08b855f-771b-4002-9555-c7d56ba510c8" (UID: "f08b855f-771b-4002-9555-c7d56ba510c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:15.117872 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.117850    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-istio-data" pod "d441e1d3-9571-4513-8404-2d6569566ba9" (UID: "d441e1d3-9571-4513-8404-2d6569566ba9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:15.117967 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.117942    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-workload-socket" pod "f08b855f-771b-4002-9555-c7d56ba510c8" (UID: "f08b855f-771b-4002-9555-c7d56ba510c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:15.118393 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.118371    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d441e1d3-9571-4513-8404-2d6569566ba9-istiod-ca-cert" pod "d441e1d3-9571-4513-8404-2d6569566ba9" (UID: "d441e1d3-9571-4513-8404-2d6569566ba9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:15.124550 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.124505    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-istio-envoy" pod "d441e1d3-9571-4513-8404-2d6569566ba9" (UID: "d441e1d3-9571-4513-8404-2d6569566ba9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:15.127612 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.127563    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d441e1d3-9571-4513-8404-2d6569566ba9-istio-token" pod "d441e1d3-9571-4513-8404-2d6569566ba9" (UID: "d441e1d3-9571-4513-8404-2d6569566ba9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:15.128826 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.128784    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08b855f-771b-4002-9555-c7d56ba510c8-istio-token" pod "f08b855f-771b-4002-9555-c7d56ba510c8" (UID: "f08b855f-771b-4002-9555-c7d56ba510c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:15.132193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.132159    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-istio-envoy" pod "f08b855f-771b-4002-9555-c7d56ba510c8" (UID: "f08b855f-771b-4002-9555-c7d56ba510c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:15.139304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.139270    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08b855f-771b-4002-9555-c7d56ba510c8-kube-api-access-nrtqd" pod "f08b855f-771b-4002-9555-c7d56ba510c8" (UID: "f08b855f-771b-4002-9555-c7d56ba510c8"). InnerVolumeSpecName "kube-api-access-nrtqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:15.140698 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.140009    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f08b855f-771b-4002-9555-c7d56ba510c8-istio-podinfo" pod "f08b855f-771b-4002-9555-c7d56ba510c8" (UID: "f08b855f-771b-4002-9555-c7d56ba510c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:15.140698 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.140160    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d441e1d3-9571-4513-8404-2d6569566ba9-istio-podinfo" pod "d441e1d3-9571-4513-8404-2d6569566ba9" (UID: "d441e1d3-9571-4513-8404-2d6569566ba9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:15.142179 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.142141    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d441e1d3-9571-4513-8404-2d6569566ba9-kube-api-access-8jmxg" pod "d441e1d3-9571-4513-8404-2d6569566ba9" (UID: "d441e1d3-9571-4513-8404-2d6569566ba9"). InnerVolumeSpecName "kube-api-access-8jmxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:15.219354 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.219322    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f08b855f-771b-4002-9555-c7d56ba510c8-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.219354 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.219354    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jmxg\" (UniqueName: \"kubernetes.io/projected/d441e1d3-9571-4513-8404-2d6569566ba9-kube-api-access-8jmxg\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.219605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.219370    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.219605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.219383    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d441e1d3-9571-4513-8404-2d6569566ba9-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.219605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.219396    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d441e1d3-9571-4513-8404-2d6569566ba9-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.219605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.219407    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.219605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.219419    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f08b855f-771b-4002-9555-c7d56ba510c8-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.219605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.219431    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrtqd\" (UniqueName: \"kubernetes.io/projected/f08b855f-771b-4002-9555-c7d56ba510c8-kube-api-access-nrtqd\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.219605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.219443    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.219605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.219455    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d441e1d3-9571-4513-8404-2d6569566ba9-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.219605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.219467    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f08b855f-771b-4002-9555-c7d56ba510c8-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.219605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.219479    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08b855f-771b-4002-9555-c7d56ba510c8-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.219605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.219491    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d441e1d3-9571-4513-8404-2d6569566ba9-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.623702 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.623353    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"
Oct 30 22:29:15.623928 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.623365    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2" event={"ID":"3935b91c-508a-4e8d-99f2-23281921afba","Type":"ContainerDied","Data":"083c3104495c24f5d0dfe435fc6de6e88d7ef3090f02bd97df0fd7627644d775"}
Oct 30 22:29:15.623928 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.623899    4889 scope.go:122] "RemoveContainer" containerID="611eb33d36595091c65fa3a504b77762a888833dd3fb07c2b191e0d2658937f4"
Oct 30 22:29:15.629668 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.629643    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"
Oct 30 22:29:15.630392 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.629701    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw" event={"ID":"d441e1d3-9571-4513-8404-2d6569566ba9","Type":"ContainerDied","Data":"e5b8a4c4b494e28de6fadf7f57dce0d25af6b8364e25c4efc7d03a7be3f6aa8f"}
Oct 30 22:29:15.632298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.632266    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8" event={"ID":"f08b855f-771b-4002-9555-c7d56ba510c8","Type":"ContainerDied","Data":"471a4c09707b03d4dc36a78f02ba4d2659fd9885d0cc31d40e7d9b6b99649c2d"}
Oct 30 22:29:15.632298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.632296    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"
Oct 30 22:29:15.637425 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.637404    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"
Oct 30 22:29:15.640501 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.638072    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"
Oct 30 22:29:15.642370 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.640807    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh" event={"ID":"89d5a590-0ac8-4be8-a791-942b739fc869","Type":"ContainerDied","Data":"7ca8f4fc49635152a0a59d652fc7bc4d0830c528d01da53705285e8eea82cb09"}
Oct 30 22:29:15.642370 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.640913    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"
Oct 30 22:29:15.642370 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.641727    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"
Oct 30 22:29:15.657350 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.657329    4889 scope.go:122] "RemoveContainer" containerID="052197b16869f136f5ff20a6390908b88fbaf7f035d1d121f3a066b2a867b073"
Oct 30 22:29:15.677827 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.677809    4889 scope.go:122] "RemoveContainer" containerID="6813827c1eac5f19ef56ed93458c0c58a68ccd529f1cd2af91606a58b0d53a4f"
Oct 30 22:29:15.688145 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.688125    4889 scope.go:122] "RemoveContainer" containerID="4a986469abd761f96f6a05beb519ef3ece3e499962f1db141cc3ac17f0a8b90c"
Oct 30 22:29:15.704128 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.704097    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"]
Oct 30 22:29:15.705994 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.705973    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:15.715762 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.715740    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-n9dj2"]
Oct 30 22:29:15.729779 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.729756    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq"]
Oct 30 22:29:15.730490 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.730475    4889 cpu_manager.go:414] "containerMap: removing container" podUID="3935b91c-508a-4e8d-99f2-23281921afba" containerName="istio-proxy"
Oct 30 22:29:15.730535 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.730491    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3935b91c-508a-4e8d-99f2-23281921afba" containerName="istio-proxy"
Oct 30 22:29:15.730535 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.730505    4889 cpu_manager.go:414] "containerMap: removing container" podUID="89d5a590-0ac8-4be8-a791-942b739fc869" containerName="istio-proxy"
Oct 30 22:29:15.730535 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.730510    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89d5a590-0ac8-4be8-a791-942b739fc869" containerName="istio-proxy"
Oct 30 22:29:15.730535 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.730525    4889 cpu_manager.go:414] "containerMap: removing container" podUID="f08b855f-771b-4002-9555-c7d56ba510c8" containerName="istio-proxy"
Oct 30 22:29:15.730535 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.730530    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f08b855f-771b-4002-9555-c7d56ba510c8" containerName="istio-proxy"
Oct 30 22:29:15.730680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.730537    4889 cpu_manager.go:414] "containerMap: removing container" podUID="08eb9eea-09b6-4149-9c58-5a252ef3062a" containerName="istio-proxy"
Oct 30 22:29:15.730680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.730541    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08eb9eea-09b6-4149-9c58-5a252ef3062a" containerName="istio-proxy"
Oct 30 22:29:15.730680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.730548    4889 cpu_manager.go:414] "containerMap: removing container" podUID="4f63c047-c985-45ed-92b5-3fed61c84385" containerName="istio-proxy"
Oct 30 22:29:15.730680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.730553    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f63c047-c985-45ed-92b5-3fed61c84385" containerName="istio-proxy"
Oct 30 22:29:15.730680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.730570    4889 cpu_manager.go:414] "containerMap: removing container" podUID="bd1e5c9d-b6f4-4326-8b97-9f0087c600c7" containerName="istio-proxy"
Oct 30 22:29:15.730680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.730574    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd1e5c9d-b6f4-4326-8b97-9f0087c600c7" containerName="istio-proxy"
Oct 30 22:29:15.730680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.730589    4889 cpu_manager.go:414] "containerMap: removing container" podUID="d441e1d3-9571-4513-8404-2d6569566ba9" containerName="istio-proxy"
Oct 30 22:29:15.730680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.730594    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d441e1d3-9571-4513-8404-2d6569566ba9" containerName="istio-proxy"
Oct 30 22:29:15.730680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.730613    4889 cpu_manager.go:414] "containerMap: removing container" podUID="408d956d-ee1c-44e8-9bc5-8d3fc72ab890" containerName="istio-proxy"
Oct 30 22:29:15.730680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.730617    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408d956d-ee1c-44e8-9bc5-8d3fc72ab890" containerName="istio-proxy"
Oct 30 22:29:15.730949 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.730714    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="3935b91c-508a-4e8d-99f2-23281921afba" containerName="istio-proxy"
Oct 30 22:29:15.730949 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.730721    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="89d5a590-0ac8-4be8-a791-942b739fc869" containerName="istio-proxy"
Oct 30 22:29:15.730949 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.730729    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="f08b855f-771b-4002-9555-c7d56ba510c8" containerName="istio-proxy"
Oct 30 22:29:15.730949 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.730736    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="08eb9eea-09b6-4149-9c58-5a252ef3062a" containerName="istio-proxy"
Oct 30 22:29:15.730949 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.730741    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f63c047-c985-45ed-92b5-3fed61c84385" containerName="istio-proxy"
Oct 30 22:29:15.730949 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.730750    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd1e5c9d-b6f4-4326-8b97-9f0087c600c7" containerName="istio-proxy"
Oct 30 22:29:15.730949 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.730757    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="d441e1d3-9571-4513-8404-2d6569566ba9" containerName="istio-proxy"
Oct 30 22:29:15.730949 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.730765    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="408d956d-ee1c-44e8-9bc5-8d3fc72ab890" containerName="istio-proxy"
Oct 30 22:29:15.736352 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.736336    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq"
Oct 30 22:29:15.750307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.750282    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-370ec2-dockercfg-dpb72\""
Oct 30 22:29:15.756124 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.756074    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq"]
Oct 30 22:29:15.772169 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.772147    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-370ec2\""
Oct 30 22:29:15.809460 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.809429    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx"]
Oct 30 22:29:15.815340 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.815304    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"]
Oct 30 22:29:15.815617 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.815586    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx"
Oct 30 22:29:15.820489 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.820466    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-zb2qv"]
Oct 30 22:29:15.830415 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.830377    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx"]
Oct 30 22:29:15.838886 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.838855    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m86lh\" (UniqueName: \"kubernetes.io/projected/ea29a485-bd51-436a-a6a4-03fc95118b57-kube-api-access-m86lh\") pod \"payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx\" (UID: \"ea29a485-bd51-436a-a6a4-03fc95118b57\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx"
Oct 30 22:29:15.839063 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.839041    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb4rw\" (UniqueName: \"kubernetes.io/projected/6a26178c-e117-4fcf-be0d-7a8fcf1ddef3-kube-api-access-tb4rw\") pod \"payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq\" (UID: \"6a26178c-e117-4fcf-be0d-7a8fcf1ddef3\") " pod="openshift-ingress/payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq"
Oct 30 22:29:15.839455 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.839432    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea29a485-bd51-436a-a6a4-03fc95118b57-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx\" (UID: \"ea29a485-bd51-436a-a6a4-03fc95118b57\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx"
Oct 30 22:29:15.839674 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.839654    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a26178c-e117-4fcf-be0d-7a8fcf1ddef3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq\" (UID: \"6a26178c-e117-4fcf-be0d-7a8fcf1ddef3\") " pod="openshift-ingress/payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq"
Oct 30 22:29:15.956302 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.951102    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:15.956302 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.952969    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a26178c-e117-4fcf-be0d-7a8fcf1ddef3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq\" (UID: \"6a26178c-e117-4fcf-be0d-7a8fcf1ddef3\") " pod="openshift-ingress/payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq"
Oct 30 22:29:15.956302 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.953135    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m86lh\" (UniqueName: \"kubernetes.io/projected/ea29a485-bd51-436a-a6a4-03fc95118b57-kube-api-access-m86lh\") pod \"payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx\" (UID: \"ea29a485-bd51-436a-a6a4-03fc95118b57\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx"
Oct 30 22:29:15.956302 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.953167    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb4rw\" (UniqueName: \"kubernetes.io/projected/6a26178c-e117-4fcf-be0d-7a8fcf1ddef3-kube-api-access-tb4rw\") pod \"payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq\" (UID: \"6a26178c-e117-4fcf-be0d-7a8fcf1ddef3\") " pod="openshift-ingress/payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq"
Oct 30 22:29:15.958646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.958567    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a26178c-e117-4fcf-be0d-7a8fcf1ddef3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq\" (UID: \"6a26178c-e117-4fcf-be0d-7a8fcf1ddef3\") " pod="openshift-ingress/payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq"
Oct 30 22:29:15.987260 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.970574    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea29a485-bd51-436a-a6a4-03fc95118b57-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx\" (UID: \"ea29a485-bd51-436a-a6a4-03fc95118b57\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx"
Oct 30 22:29:15.987260 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:15.971671    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea29a485-bd51-436a-a6a4-03fc95118b57-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx\" (UID: \"ea29a485-bd51-436a-a6a4-03fc95118b57\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx"
Oct 30 22:29:16.073375 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.073327    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m86lh\" (UniqueName: \"kubernetes.io/projected/ea29a485-bd51-436a-a6a4-03fc95118b57-kube-api-access-m86lh\") pod \"payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx\" (UID: \"ea29a485-bd51-436a-a6a4-03fc95118b57\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx"
Oct 30 22:29:16.105116 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.105087    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"]
Oct 30 22:29:16.113692 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.113621    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.132009 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.131979    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-510f80-openshift-default-dockercfg-w2m4c\""
Oct 30 22:29:16.138270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.137988    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb4rw\" (UniqueName: \"kubernetes.io/projected/6a26178c-e117-4fcf-be0d-7a8fcf1ddef3-kube-api-access-tb4rw\") pod \"payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq\" (UID: \"6a26178c-e117-4fcf-be0d-7a8fcf1ddef3\") " pod="openshift-ingress/payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq"
Oct 30 22:29:16.142609 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.142542    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"]
Oct 30 22:29:16.145706 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.145682    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx"
Oct 30 22:29:16.163574 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.161829    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-2t4b8"]
Oct 30 22:29:16.200796 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.200755    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65498\" (UniqueName: \"kubernetes.io/projected/666b6202-2fcf-4d36-8098-deb9c3fd3718-kube-api-access-65498\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.201016 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.200986    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-credential-socket\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.201102 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.201056    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-token\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.201156 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.201132    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-workload-socket\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.201206 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.201185    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-data\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.201370 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.201335    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-podinfo\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.201447 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.201395    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-workload-certs\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.201447 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.201432    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666b6202-2fcf-4d36-8098-deb9c3fd3718-istiod-ca-cert\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.201559 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.201507    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-envoy\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.304345 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.304284    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-podinfo\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.310563 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.310529    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-workload-certs\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.311375 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.311349    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666b6202-2fcf-4d36-8098-deb9c3fd3718-istiod-ca-cert\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.319478 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.319432    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-envoy\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.326481 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.326449    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65498\" (UniqueName: \"kubernetes.io/projected/666b6202-2fcf-4d36-8098-deb9c3fd3718-kube-api-access-65498\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.335419 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.327158    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-credential-socket\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.337306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.337284    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-token\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.337777 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.337759    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-workload-socket\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.338339 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.338321    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-data\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.338889 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.311197    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-workload-certs\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.339002 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.337153    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-credential-socket\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.339104 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.338123    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-workload-socket\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.339185 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.319151    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666b6202-2fcf-4d36-8098-deb9c3fd3718-istiod-ca-cert\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.341525 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.338650    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-data\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.367474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.367065    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq"
Oct 30 22:29:16.373187 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.373126    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-envoy\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.408284 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.383109    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65498\" (UniqueName: \"kubernetes.io/projected/666b6202-2fcf-4d36-8098-deb9c3fd3718-kube-api-access-65498\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.408284 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.386418    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:16.425167 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.425076    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-podinfo\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.428295 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.428175    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-token\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.518383 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.509291    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"]
Oct 30 22:29:16.518383 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.517830    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:16.526909 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.526871    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"]
Oct 30 22:29:16.532141 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.532090    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-c9dkh"]
Oct 30 22:29:16.538329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.538298    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"]
Oct 30 22:29:16.542378 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.542355    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-7wsq8"]
Oct 30 22:29:16.557328 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.552976    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"]
Oct 30 22:29:16.562448 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.562245    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-2w4gw"]
Oct 30 22:29:16.568637 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.568611    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"]
Oct 30 22:29:16.582109 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.578435    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-m6s8k"]
Oct 30 22:29:16.780671 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.780638    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08eb9eea-09b6-4149-9c58-5a252ef3062a" path="/var/lib/kubelet/pods/08eb9eea-09b6-4149-9c58-5a252ef3062a/volumes"
Oct 30 22:29:16.781156 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.781140    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3935b91c-508a-4e8d-99f2-23281921afba" path="/var/lib/kubelet/pods/3935b91c-508a-4e8d-99f2-23281921afba/volumes"
Oct 30 22:29:16.781614 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.781598    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="408d956d-ee1c-44e8-9bc5-8d3fc72ab890" path="/var/lib/kubelet/pods/408d956d-ee1c-44e8-9bc5-8d3fc72ab890/volumes"
Oct 30 22:29:16.782375 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.782354    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89d5a590-0ac8-4be8-a791-942b739fc869" path="/var/lib/kubelet/pods/89d5a590-0ac8-4be8-a791-942b739fc869/volumes"
Oct 30 22:29:16.782892 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.782875    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd1e5c9d-b6f4-4326-8b97-9f0087c600c7" path="/var/lib/kubelet/pods/bd1e5c9d-b6f4-4326-8b97-9f0087c600c7/volumes"
Oct 30 22:29:16.783772 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.783749    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d441e1d3-9571-4513-8404-2d6569566ba9" path="/var/lib/kubelet/pods/d441e1d3-9571-4513-8404-2d6569566ba9/volumes"
Oct 30 22:29:16.784692 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.784673    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f08b855f-771b-4002-9555-c7d56ba510c8" path="/var/lib/kubelet/pods/f08b855f-771b-4002-9555-c7d56ba510c8/volumes"
Oct 30 22:29:16.798612 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:29:16.798556    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea29a485_bd51_436a_a6a4_03fc95118b57.slice/crio-5b72ac43b2ebb1b2e7a576242f4b2b2fd3bb95a17d6b9b1111e457f299f4dfcc WatchSource:0}: Error finding container 5b72ac43b2ebb1b2e7a576242f4b2b2fd3bb95a17d6b9b1111e457f299f4dfcc: Status 404 returned error can't find the container with id 5b72ac43b2ebb1b2e7a576242f4b2b2fd3bb95a17d6b9b1111e457f299f4dfcc
Oct 30 22:29:16.799530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.799510    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx"]
Oct 30 22:29:16.839625 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:29:16.839584    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a26178c_e117_4fcf_be0d_7a8fcf1ddef3.slice/crio-b71f8b221464c427f93e1db436137edb4b55f352787cf755d15a3fb5145f7a9e WatchSource:0}: Error finding container b71f8b221464c427f93e1db436137edb4b55f352787cf755d15a3fb5145f7a9e: Status 404 returned error can't find the container with id b71f8b221464c427f93e1db436137edb4b55f352787cf755d15a3fb5145f7a9e
Oct 30 22:29:16.840095 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.840073    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq"]
Oct 30 22:29:16.885622 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.885593    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"]
Oct 30 22:29:16.886752 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:29:16.886725    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod666b6202_2fcf_4d36_8098_deb9c3fd3718.slice/crio-fe3259e4834d022a7fccb4926223152a8c4398adf64e8191f550231bfafd7b7b WatchSource:0}: Error finding container fe3259e4834d022a7fccb4926223152a8c4398adf64e8191f550231bfafd7b7b: Status 404 returned error can't find the container with id fe3259e4834d022a7fccb4926223152a8c4398adf64e8191f550231bfafd7b7b
Oct 30 22:29:16.890142 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.890109    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:29:16.890244 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.890178    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:29:16.890244 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:16.890204    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:29:17.410480 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.410443    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:17.665695 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.665603    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx" event={"ID":"ea29a485-bd51-436a-a6a4-03fc95118b57","Type":"ContainerStarted","Data":"0cb162f0eb6aa5ebb9faeadc388b86b4190fd32e5148dba14fed18a05412e637"}
Oct 30 22:29:17.665695 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.665654    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx" event={"ID":"ea29a485-bd51-436a-a6a4-03fc95118b57","Type":"ContainerStarted","Data":"5b72ac43b2ebb1b2e7a576242f4b2b2fd3bb95a17d6b9b1111e457f299f4dfcc"}
Oct 30 22:29:17.665930 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.665831    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx"
Oct 30 22:29:17.667834 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.667803    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns" event={"ID":"666b6202-2fcf-4d36-8098-deb9c3fd3718","Type":"ContainerStarted","Data":"1fe48a81d92e6664261d0400f1a1cd497211eaa83ed57ea704bdc7872ece7263"}
Oct 30 22:29:17.667945 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.667862    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns" event={"ID":"666b6202-2fcf-4d36-8098-deb9c3fd3718","Type":"ContainerStarted","Data":"fe3259e4834d022a7fccb4926223152a8c4398adf64e8191f550231bfafd7b7b"}
Oct 30 22:29:17.670423 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.670400    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq" event={"ID":"6a26178c-e117-4fcf-be0d-7a8fcf1ddef3","Type":"ContainerStarted","Data":"0ea3df77d9f37b970123dc7112573ce5cda03646d8293fecab761309b221cc9f"}
Oct 30 22:29:17.670514 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.670444    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq" event={"ID":"6a26178c-e117-4fcf-be0d-7a8fcf1ddef3","Type":"ContainerStarted","Data":"b71f8b221464c427f93e1db436137edb4b55f352787cf755d15a3fb5145f7a9e"}
Oct 30 22:29:17.670514 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.670475    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq"
Oct 30 22:29:17.701341 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.701301    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx" podStartSLOduration=2.701289609 podStartE2EDuration="2.701289609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:17.700369839 +0000 UTC m=+1343.056736154" watchObservedRunningTime="2026-10-30 22:29:17.701289609 +0000 UTC m=+1343.057655923"
Oct 30 22:29:17.747313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.747264    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns" podStartSLOduration=2.747248542 podStartE2EDuration="2.747248542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:17.744448638 +0000 UTC m=+1343.100814953" watchObservedRunningTime="2026-10-30 22:29:17.747248542 +0000 UTC m=+1343.103614852"
Oct 30 22:29:17.760404 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.760376    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"]
Oct 30 22:29:17.765289 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.765265    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:17.791183 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.785097    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-0ff5b0-openshift-default-dockercfg-2wvx5\""
Oct 30 22:29:17.791183 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.786878    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"]
Oct 30 22:29:17.791183 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.789986    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq" podStartSLOduration=2.789970306 podStartE2EDuration="2.789970306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:17.784923512 +0000 UTC m=+1343.141289833" watchObservedRunningTime="2026-10-30 22:29:17.789970306 +0000 UTC m=+1343.146336621"
Oct 30 22:29:17.797869 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.797830    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-workload-socket\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:17.798024 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.797910    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wcdl\" (UniqueName: \"kubernetes.io/projected/7cfa769f-0310-4749-a929-646c9ab9bea5-kube-api-access-6wcdl\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:17.798209 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.798179    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-podinfo\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:17.798393 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.798348    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-data\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:17.798393 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.798377    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-token\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:17.798541 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.798520    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-workload-certs\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:17.798592 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.798558    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cfa769f-0310-4749-a929-646c9ab9bea5-istiod-ca-cert\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:17.798639 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.798611    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-credential-socket\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:17.798692 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.798644    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-envoy\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:17.900348 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.900312    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-podinfo\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:17.900540 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.900427    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-data\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:17.900540 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.900445    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-token\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:17.900540 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.900484    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-workload-certs\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:17.900540 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.900505    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cfa769f-0310-4749-a929-646c9ab9bea5-istiod-ca-cert\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:17.900758 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.900558    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-credential-socket\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:17.900758 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.900592    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-envoy\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:17.900758 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.900625    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-workload-socket\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:17.900758 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.900660    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wcdl\" (UniqueName: \"kubernetes.io/projected/7cfa769f-0310-4749-a929-646c9ab9bea5-kube-api-access-6wcdl\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:17.901210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.901181    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-workload-certs\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:17.901390 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.901211    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-data\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:17.901493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.901457    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-credential-socket\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:17.901738 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.901713    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-workload-socket\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:17.903060 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.903037    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cfa769f-0310-4749-a929-646c9ab9bea5-istiod-ca-cert\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:17.903283 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.903264    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-podinfo\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:17.903468 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.903452    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-envoy\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:17.907529 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.907460    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:17.967555 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.965818    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wcdl\" (UniqueName: \"kubernetes.io/projected/7cfa769f-0310-4749-a929-646c9ab9bea5-kube-api-access-6wcdl\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:17.996245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:17.992521    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-token\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:18.103259 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:18.102600    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:18.113253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:18.105832    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:18.294249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:18.288434    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:18.522256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:18.522005    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:18.522256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:18.522131    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:29:18.522256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:18.522173    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns" podUID="666b6202-2fcf-4d36-8098-deb9c3fd3718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 22:29:18.780252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:18.772951    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:18.881297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:18.876569    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"]
Oct 30 22:29:18.894276 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:29:18.894183    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cfa769f_0310_4749_a929_646c9ab9bea5.slice/crio-483e155a808ab15edf6a8cc6b59ea6156cbc40645c6a07b9567b8de424d882f2 WatchSource:0}: Error finding container 483e155a808ab15edf6a8cc6b59ea6156cbc40645c6a07b9567b8de424d882f2: Status 404 returned error can't find the container with id 483e155a808ab15edf6a8cc6b59ea6156cbc40645c6a07b9567b8de424d882f2
Oct 30 22:29:18.897801 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:18.897763    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:29:18.897894 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:18.897868    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:29:18.897934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:18.897907    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:29:19.532258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:19.523281    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:29:19.532258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:19.523341    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns" podUID="666b6202-2fcf-4d36-8098-deb9c3fd3718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 22:29:19.693032 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:19.692994    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h" event={"ID":"7cfa769f-0310-4749-a929-646c9ab9bea5","Type":"ContainerStarted","Data":"fcd12a6635b21bdca58d709888fee4cd88f8890fa114044e2004c2c4ef80c411"}
Oct 30 22:29:19.693207 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:19.693046    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h" event={"ID":"7cfa769f-0310-4749-a929-646c9ab9bea5","Type":"ContainerStarted","Data":"483e155a808ab15edf6a8cc6b59ea6156cbc40645c6a07b9567b8de424d882f2"}
Oct 30 22:29:19.761418 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:19.760182    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h" podStartSLOduration=2.760164795 podStartE2EDuration="2.760164795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:19.756655875 +0000 UTC m=+1345.113022189" watchObservedRunningTime="2026-10-30 22:29:19.760164795 +0000 UTC m=+1345.116531109"
Oct 30 22:29:20.109053 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:20.106587    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:20.147253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:20.146555    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:20.530679 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:20.530639    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:29:20.531202 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:20.531170    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns" podUID="666b6202-2fcf-4d36-8098-deb9c3fd3718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 22:29:20.656275 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:20.654588    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:20.729248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:20.722417    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:20.736622 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:20.736588    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:29:21.267340 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:21.263254    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:21.531308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:21.530625    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:29:21.531308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:21.530687    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns" podUID="666b6202-2fcf-4d36-8098-deb9c3fd3718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 22:29:22.276510 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:22.276479    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:22.446083 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:22.445930    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:22.530305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:22.524848    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:29:22.530305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:22.524910    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns" podUID="666b6202-2fcf-4d36-8098-deb9c3fd3718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 22:29:22.636313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:22.634096    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:22.825332 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:22.811628    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:23.034881 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:23.034842    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:23.237305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:23.235490    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:23.519572 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:23.519537    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:29:23.520023 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:23.519606    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns" podUID="666b6202-2fcf-4d36-8098-deb9c3fd3718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 22:29:24.136869 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:24.136837    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:24.252858 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:24.252829    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt"]
Oct 30 22:29:24.257521 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:24.257502    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt"
Oct 30 22:29:24.277338 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:24.277306    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt"]
Oct 30 22:29:24.281117 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:24.281078    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-0ff5b0\""
Oct 30 22:29:24.281284 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:24.281130    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-0ff5b0-dockercfg-pklx2\""
Oct 30 22:29:24.300601 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:24.300572    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:24.412575 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:24.412471    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b2f029-b001-48d1-8c60-237096ebe16e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt\" (UID: \"63b2f029-b001-48d1-8c60-237096ebe16e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt"
Oct 30 22:29:24.413048 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:24.413021    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65wt6\" (UniqueName: \"kubernetes.io/projected/63b2f029-b001-48d1-8c60-237096ebe16e-kube-api-access-65wt6\") pod \"payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt\" (UID: \"63b2f029-b001-48d1-8c60-237096ebe16e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt"
Oct 30 22:29:24.542261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:24.526791    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b2f029-b001-48d1-8c60-237096ebe16e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt\" (UID: \"63b2f029-b001-48d1-8c60-237096ebe16e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt"
Oct 30 22:29:24.563199 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:24.563158    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65wt6\" (UniqueName: \"kubernetes.io/projected/63b2f029-b001-48d1-8c60-237096ebe16e-kube-api-access-65wt6\") pod \"payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt\" (UID: \"63b2f029-b001-48d1-8c60-237096ebe16e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt"
Oct 30 22:29:24.564149 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:24.533594    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b2f029-b001-48d1-8c60-237096ebe16e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt\" (UID: \"63b2f029-b001-48d1-8c60-237096ebe16e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt"
Oct 30 22:29:24.625129 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:24.625098    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65wt6\" (UniqueName: \"kubernetes.io/projected/63b2f029-b001-48d1-8c60-237096ebe16e-kube-api-access-65wt6\") pod \"payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt\" (UID: \"63b2f029-b001-48d1-8c60-237096ebe16e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt"
Oct 30 22:29:24.632729 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:24.632689    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52"]
Oct 30 22:29:24.639452 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:24.639425    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:24.640777 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:24.639948    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52"
Oct 30 22:29:24.640777 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:24.640727    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:24.641810 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:24.641785    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52"]
Oct 30 22:29:24.642011 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:24.641991    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:29:24.776339 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:24.776242    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwhrc\" (UniqueName: \"kubernetes.io/projected/9683180a-1f65-48bb-a62c-c78cef5799a3-kube-api-access-kwhrc\") pod \"payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52\" (UID: \"9683180a-1f65-48bb-a62c-c78cef5799a3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52"
Oct 30 22:29:24.776339 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:24.776311    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9683180a-1f65-48bb-a62c-c78cef5799a3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52\" (UID: \"9683180a-1f65-48bb-a62c-c78cef5799a3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52"
Oct 30 22:29:24.875885 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:24.875856    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt"
Oct 30 22:29:24.878641 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:24.878535    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwhrc\" (UniqueName: \"kubernetes.io/projected/9683180a-1f65-48bb-a62c-c78cef5799a3-kube-api-access-kwhrc\") pod \"payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52\" (UID: \"9683180a-1f65-48bb-a62c-c78cef5799a3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52"
Oct 30 22:29:24.878641 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:24.878627    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9683180a-1f65-48bb-a62c-c78cef5799a3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52\" (UID: \"9683180a-1f65-48bb-a62c-c78cef5799a3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52"
Oct 30 22:29:24.879951 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:24.879809    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9683180a-1f65-48bb-a62c-c78cef5799a3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52\" (UID: \"9683180a-1f65-48bb-a62c-c78cef5799a3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52"
Oct 30 22:29:24.897033 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:24.897001    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwhrc\" (UniqueName: \"kubernetes.io/projected/9683180a-1f65-48bb-a62c-c78cef5799a3-kube-api-access-kwhrc\") pod \"payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52\" (UID: \"9683180a-1f65-48bb-a62c-c78cef5799a3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52"
Oct 30 22:29:24.985160 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:24.984442    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52"
Oct 30 22:29:25.208259 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:25.204595    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:25.281531 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:25.279566    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt"]
Oct 30 22:29:25.316087 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:25.316062    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52"]
Oct 30 22:29:25.317672 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:29:25.317651    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9683180a_1f65_48bb_a62c_c78cef5799a3.slice/crio-e03366a02d5b1fd31a33ab6b69505e4d9daec3bda4811c91cc3d4edffe17bfcd WatchSource:0}: Error finding container e03366a02d5b1fd31a33ab6b69505e4d9daec3bda4811c91cc3d4edffe17bfcd: Status 404 returned error can't find the container with id e03366a02d5b1fd31a33ab6b69505e4d9daec3bda4811c91cc3d4edffe17bfcd
Oct 30 22:29:25.803557 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:25.803517    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52" event={"ID":"9683180a-1f65-48bb-a62c-c78cef5799a3","Type":"ContainerStarted","Data":"3250ae08a08187eccf3a2f33c7b06aeb17c5b15ced73b0b490efe86572b354ea"}
Oct 30 22:29:25.804350 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:25.803592    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52"
Oct 30 22:29:25.804350 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:25.803614    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52" event={"ID":"9683180a-1f65-48bb-a62c-c78cef5799a3","Type":"ContainerStarted","Data":"e03366a02d5b1fd31a33ab6b69505e4d9daec3bda4811c91cc3d4edffe17bfcd"}
Oct 30 22:29:25.805203 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:25.805174    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt" event={"ID":"63b2f029-b001-48d1-8c60-237096ebe16e","Type":"ContainerStarted","Data":"2418049e857e65cd08a191d3598901910c138abf046ca4c7101712edd89d5f03"}
Oct 30 22:29:25.805343 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:25.805242    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt" event={"ID":"63b2f029-b001-48d1-8c60-237096ebe16e","Type":"ContainerStarted","Data":"0d37f7f77a96325afbdfe767e5818d3d4d7696eca9e0714e5268554a10e0859a"}
Oct 30 22:29:25.805343 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:25.805280    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt"
Oct 30 22:29:25.843428 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:25.843379    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52" podStartSLOduration=1.8433665289999999 podStartE2EDuration="1.843366529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:25.839203351 +0000 UTC m=+1351.195569673" watchObservedRunningTime="2026-10-30 22:29:25.843366529 +0000 UTC m=+1351.199732868"
Oct 30 22:29:25.877030 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:25.876981    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt" podStartSLOduration=1.876969348 podStartE2EDuration="1.876969348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:25.876292648 +0000 UTC m=+1351.232658961" watchObservedRunningTime="2026-10-30 22:29:25.876969348 +0000 UTC m=+1351.233335698"
Oct 30 22:29:27.134965 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:27.134917    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:27.453501 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:27.453408    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:27.627647 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:27.627614    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:28.686368 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:28.686199    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq"
Oct 30 22:29:28.686970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:28.686727    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx"
Oct 30 22:29:28.774145 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:28.774110    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:29.152250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:29.151377    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:29.320261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:29.315517    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:29.790521 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:29.790488    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:29.986254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:29.976749    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:31.817264 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:31.816503    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:33.522565 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:33.520967    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:34.043094 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:34.043052    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:35.432793 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:35.432750    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:36.815925 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:36.815898    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52"
Oct 30 22:29:36.816383 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:36.816343    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt"
Oct 30 22:29:37.316094 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:37.316057    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:37.487626 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:37.487587    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:38.143845 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:38.143814    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:38.200610 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:38.200584    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn"]
Oct 30 22:29:38.206382 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:38.206362    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn"
Oct 30 22:29:38.227456 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:38.227433    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn"]
Oct 30 22:29:38.231129 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:38.231100    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2949c143\""
Oct 30 22:29:38.274974 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:38.274945    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqxhr\" (UniqueName: \"kubernetes.io/projected/440bb07c-dd47-45ed-a095-f4247e739d36-kube-api-access-mqxhr\") pod \"payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn\" (UID: \"440bb07c-dd47-45ed-a095-f4247e739d36\") " pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn"
Oct 30 22:29:38.275210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:38.275189    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440bb07c-dd47-45ed-a095-f4247e739d36-plugins-config-volume\") pod \"payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn\" (UID: \"440bb07c-dd47-45ed-a095-f4247e739d36\") " pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn"
Oct 30 22:29:38.283487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:38.283465    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2949c143-dockercfg-mmm8r\""
Oct 30 22:29:38.421300 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:38.418494    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440bb07c-dd47-45ed-a095-f4247e739d36-plugins-config-volume\") pod \"payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn\" (UID: \"440bb07c-dd47-45ed-a095-f4247e739d36\") " pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn"
Oct 30 22:29:38.421300 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:38.418795    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqxhr\" (UniqueName: \"kubernetes.io/projected/440bb07c-dd47-45ed-a095-f4247e739d36-kube-api-access-mqxhr\") pod \"payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn\" (UID: \"440bb07c-dd47-45ed-a095-f4247e739d36\") " pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn"
Oct 30 22:29:38.444270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:38.442671    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:38.516248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:38.515461    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440bb07c-dd47-45ed-a095-f4247e739d36-plugins-config-volume\") pod \"payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn\" (UID: \"440bb07c-dd47-45ed-a095-f4247e739d36\") " pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn"
Oct 30 22:29:38.681357 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:38.662016    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqxhr\" (UniqueName: \"kubernetes.io/projected/440bb07c-dd47-45ed-a095-f4247e739d36-kube-api-access-mqxhr\") pod \"payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn\" (UID: \"440bb07c-dd47-45ed-a095-f4247e739d36\") " pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn"
Oct 30 22:29:38.823274 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:38.823239    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn"
Oct 30 22:29:39.050061 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:39.050023    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"]
Oct 30 22:29:39.050297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:39.050255    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t" podUID="32e978e1-1585-4947-8975-fe20d4942bc5" containerName="istio-proxy" containerID="cri-o://2bde103a7436ca298adbd1ab5c82638f9ac58bd4b7aee7b0a3fd1e41fe9cef14" gracePeriod=30
Oct 30 22:29:39.061306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:39.061274    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"]
Oct 30 22:29:39.061745 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:39.061563    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh" podUID="69ae6469-721f-4879-bd5f-132fbc422276" containerName="istio-proxy" containerID="cri-o://b3ebf97f820d7239599f3a535efaddba67c88d9055561d5a53909d00eeee2aae" gracePeriod=30
Oct 30 22:29:39.066628 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:39.066209    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"]
Oct 30 22:29:39.066628 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:39.066453    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr" podUID="6ad3d3d8-c760-4b47-925b-aa2c80d43883" containerName="istio-proxy" containerID="cri-o://ffae4b7cddb321b84e337d0db558a30d8b26430db974accef70a3c4ce893d01c" gracePeriod=30
Oct 30 22:29:39.129655 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:29:39.129620    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440bb07c_dd47_45ed_a095_f4247e739d36.slice/crio-c86af3ff6a3e6f5ea0943e99a844ead328f00c1463239c6a0088b706aa2c4a7d WatchSource:0}: Error finding container c86af3ff6a3e6f5ea0943e99a844ead328f00c1463239c6a0088b706aa2c4a7d: Status 404 returned error can't find the container with id c86af3ff6a3e6f5ea0943e99a844ead328f00c1463239c6a0088b706aa2c4a7d
Oct 30 22:29:39.129816 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:39.129710    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn"]
Oct 30 22:29:39.412297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:39.411181    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:39.958318 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:39.958280    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn" event={"ID":"440bb07c-dd47-45ed-a095-f4247e739d36","Type":"ContainerStarted","Data":"28bda0e483290039c099bb818b4514acae755c6abfaa7fbfd47c318994afa147"}
Oct 30 22:29:39.958491 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:39.958342    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn" event={"ID":"440bb07c-dd47-45ed-a095-f4247e739d36","Type":"ContainerStarted","Data":"c86af3ff6a3e6f5ea0943e99a844ead328f00c1463239c6a0088b706aa2c4a7d"}
Oct 30 22:29:39.958491 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:39.958364    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn"
Oct 30 22:29:39.999623 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:39.999571    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn" podStartSLOduration=1.999557638 podStartE2EDuration="1.999557638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:39.994894283 +0000 UTC m=+1365.351260599" watchObservedRunningTime="2026-10-30 22:29:39.999557638 +0000 UTC m=+1365.355923951"
Oct 30 22:29:40.366015 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:29:40.365950    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:29:41.241245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:41.237057    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:42.991256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:42.988736    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:44.281788 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.281740    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:44.377145 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.377117    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:29:44.426741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.426700    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:29:44.431112 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.431091    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:29:44.454914 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.454876    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istiod-ca-cert\") pod \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") "
Oct 30 22:29:44.455101 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.454966    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-envoy\") pod \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") "
Oct 30 22:29:44.455101 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.455035    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-data\") pod \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") "
Oct 30 22:29:44.455209 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.455099    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-credential-socket\") pod \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") "
Oct 30 22:29:44.455209 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.455142    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-workload-socket\") pod \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") "
Oct 30 22:29:44.455361 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.455279    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-token\" (UniqueName: \"kubernetes.io/projected/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-token\") pod \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") "
Oct 30 22:29:44.455361 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.455320    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ad3d3d8-c760-4b47-925b-aa2c80d43883-kube-api-access-jdbdc\" (UniqueName: \"kubernetes.io/projected/6ad3d3d8-c760-4b47-925b-aa2c80d43883-kube-api-access-jdbdc\") pod \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") "
Oct 30 22:29:44.455453 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.455403    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-podinfo\") pod \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") "
Oct 30 22:29:44.455521 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.455501    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-workload-certs\") pod \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\" (UID: \"6ad3d3d8-c760-4b47-925b-aa2c80d43883\") "
Oct 30 22:29:44.456630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.456602    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-workload-certs" pod "6ad3d3d8-c760-4b47-925b-aa2c80d43883" (UID: "6ad3d3d8-c760-4b47-925b-aa2c80d43883"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:44.456968 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.456945    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istiod-ca-cert" pod "6ad3d3d8-c760-4b47-925b-aa2c80d43883" (UID: "6ad3d3d8-c760-4b47-925b-aa2c80d43883"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:44.458157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.458128    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-data" pod "6ad3d3d8-c760-4b47-925b-aa2c80d43883" (UID: "6ad3d3d8-c760-4b47-925b-aa2c80d43883"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:44.458424 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.458400    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-workload-socket" pod "6ad3d3d8-c760-4b47-925b-aa2c80d43883" (UID: "6ad3d3d8-c760-4b47-925b-aa2c80d43883"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:44.459452 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.459415    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-credential-socket" pod "6ad3d3d8-c760-4b47-925b-aa2c80d43883" (UID: "6ad3d3d8-c760-4b47-925b-aa2c80d43883"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:44.460391 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.460361    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad3d3d8-c760-4b47-925b-aa2c80d43883-kube-api-access-jdbdc" pod "6ad3d3d8-c760-4b47-925b-aa2c80d43883" (UID: "6ad3d3d8-c760-4b47-925b-aa2c80d43883"). InnerVolumeSpecName "kube-api-access-jdbdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:44.460866 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.460841    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-token" pod "6ad3d3d8-c760-4b47-925b-aa2c80d43883" (UID: "6ad3d3d8-c760-4b47-925b-aa2c80d43883"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:44.462445 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.462421    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-envoy" pod "6ad3d3d8-c760-4b47-925b-aa2c80d43883" (UID: "6ad3d3d8-c760-4b47-925b-aa2c80d43883"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:44.462890 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.462865    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-podinfo" pod "6ad3d3d8-c760-4b47-925b-aa2c80d43883" (UID: "6ad3d3d8-c760-4b47-925b-aa2c80d43883"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:44.495537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.495501    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:44.557295 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.557200    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-credential-socket\") pod \"69ae6469-721f-4879-bd5f-132fbc422276\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") "
Oct 30 22:29:44.557476 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.557349    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69ae6469-721f-4879-bd5f-132fbc422276-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ae6469-721f-4879-bd5f-132fbc422276-istiod-ca-cert\") pod \"69ae6469-721f-4879-bd5f-132fbc422276\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") "
Oct 30 22:29:44.557476 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.557380    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-workload-certs\") pod \"32e978e1-1585-4947-8975-fe20d4942bc5\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") "
Oct 30 22:29:44.557476 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.557420    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-workload-socket\") pod \"69ae6469-721f-4879-bd5f-132fbc422276\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") "
Oct 30 22:29:44.557476 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.557449    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69ae6469-721f-4879-bd5f-132fbc422276-istio-token\" (UniqueName: \"kubernetes.io/projected/69ae6469-721f-4879-bd5f-132fbc422276-istio-token\") pod \"69ae6469-721f-4879-bd5f-132fbc422276\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") "
Oct 30 22:29:44.557706 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.557489    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69ae6469-721f-4879-bd5f-132fbc422276-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ae6469-721f-4879-bd5f-132fbc422276-istio-podinfo\") pod \"69ae6469-721f-4879-bd5f-132fbc422276\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") "
Oct 30 22:29:44.557706 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.557514    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-credential-socket" pod "69ae6469-721f-4879-bd5f-132fbc422276" (UID: "69ae6469-721f-4879-bd5f-132fbc422276"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:44.557706 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.557598    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32e978e1-1585-4947-8975-fe20d4942bc5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32e978e1-1585-4947-8975-fe20d4942bc5-istiod-ca-cert\") pod \"32e978e1-1585-4947-8975-fe20d4942bc5\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") "
Oct 30 22:29:44.557706 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.557627    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-workload-certs" pod "32e978e1-1585-4947-8975-fe20d4942bc5" (UID: "32e978e1-1585-4947-8975-fe20d4942bc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:44.557706 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.557679    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32e978e1-1585-4947-8975-fe20d4942bc5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32e978e1-1585-4947-8975-fe20d4942bc5-istio-podinfo\") pod \"32e978e1-1585-4947-8975-fe20d4942bc5\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") "
Oct 30 22:29:44.557963 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.557717    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-workload-certs\") pod \"69ae6469-721f-4879-bd5f-132fbc422276\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") "
Oct 30 22:29:44.557963 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.557754    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32e978e1-1585-4947-8975-fe20d4942bc5-kube-api-access-w6lhn\" (UniqueName: \"kubernetes.io/projected/32e978e1-1585-4947-8975-fe20d4942bc5-kube-api-access-w6lhn\") pod \"32e978e1-1585-4947-8975-fe20d4942bc5\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") "
Oct 30 22:29:44.557963 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.557789    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-workload-socket\") pod \"32e978e1-1585-4947-8975-fe20d4942bc5\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") "
Oct 30 22:29:44.557963 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.557828    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69ae6469-721f-4879-bd5f-132fbc422276-kube-api-access-8mfmw\" (UniqueName: \"kubernetes.io/projected/69ae6469-721f-4879-bd5f-132fbc422276-kube-api-access-8mfmw\") pod \"69ae6469-721f-4879-bd5f-132fbc422276\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") "
Oct 30 22:29:44.557963 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.557838    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69ae6469-721f-4879-bd5f-132fbc422276-istiod-ca-cert" pod "69ae6469-721f-4879-bd5f-132fbc422276" (UID: "69ae6469-721f-4879-bd5f-132fbc422276"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:44.557963 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.557912    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-istio-data\") pod \"32e978e1-1585-4947-8975-fe20d4942bc5\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") "
Oct 30 22:29:44.557963 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.557950    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-istio-envoy\") pod \"32e978e1-1585-4947-8975-fe20d4942bc5\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") "
Oct 30 22:29:44.557963 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.557958    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-workload-certs" pod "69ae6469-721f-4879-bd5f-132fbc422276" (UID: "69ae6469-721f-4879-bd5f-132fbc422276"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:44.558443 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.558141    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-workload-socket" pod "32e978e1-1585-4947-8975-fe20d4942bc5" (UID: "32e978e1-1585-4947-8975-fe20d4942bc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:44.558443 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.558334    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-workload-socket" pod "69ae6469-721f-4879-bd5f-132fbc422276" (UID: "69ae6469-721f-4879-bd5f-132fbc422276"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:44.558670 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.558642    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32e978e1-1585-4947-8975-fe20d4942bc5-istiod-ca-cert" pod "32e978e1-1585-4947-8975-fe20d4942bc5" (UID: "32e978e1-1585-4947-8975-fe20d4942bc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:44.558750 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.558713    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-istio-data" pod "32e978e1-1585-4947-8975-fe20d4942bc5" (UID: "32e978e1-1585-4947-8975-fe20d4942bc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:44.559387 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.559356    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-istio-data\") pod \"69ae6469-721f-4879-bd5f-132fbc422276\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") "
Oct 30 22:29:44.559517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.559429    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32e978e1-1585-4947-8975-fe20d4942bc5-istio-token\" (UniqueName: \"kubernetes.io/projected/32e978e1-1585-4947-8975-fe20d4942bc5-istio-token\") pod \"32e978e1-1585-4947-8975-fe20d4942bc5\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") "
Oct 30 22:29:44.559517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.559476    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-credential-socket\") pod \"32e978e1-1585-4947-8975-fe20d4942bc5\" (UID: \"32e978e1-1585-4947-8975-fe20d4942bc5\") "
Oct 30 22:29:44.559680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.559518    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-istio-envoy\") pod \"69ae6469-721f-4879-bd5f-132fbc422276\" (UID: \"69ae6469-721f-4879-bd5f-132fbc422276\") "
Oct 30 22:29:44.559736 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.559720    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-istio-data" pod "69ae6469-721f-4879-bd5f-132fbc422276" (UID: "69ae6469-721f-4879-bd5f-132fbc422276"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:44.560437 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.560378    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:44.560437 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.560400    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:44.560437 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.560412    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:44.560437 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.560425    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdbdc\" (UniqueName: \"kubernetes.io/projected/6ad3d3d8-c760-4b47-925b-aa2c80d43883-kube-api-access-jdbdc\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:44.560821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.560441    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:44.560821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.560455    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:44.560821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.560468    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:44.560821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.560479    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ae6469-721f-4879-bd5f-132fbc422276-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:44.560821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.560490    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:44.560821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.560501    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:44.560821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.560512    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:44.560821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.560523    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:44.560821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.560534    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32e978e1-1585-4947-8975-fe20d4942bc5-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:44.560821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.560552    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:44.560821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.560563    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:44.560821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.560576    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:44.560821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.560587    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad3d3d8-c760-4b47-925b-aa2c80d43883-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:44.560821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.560600    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:44.561511 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.560949    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ae6469-721f-4879-bd5f-132fbc422276-kube-api-access-8mfmw" pod "69ae6469-721f-4879-bd5f-132fbc422276" (UID: "69ae6469-721f-4879-bd5f-132fbc422276"). InnerVolumeSpecName "kube-api-access-8mfmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:44.561511 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.561217    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-credential-socket" pod "32e978e1-1585-4947-8975-fe20d4942bc5" (UID: "32e978e1-1585-4947-8975-fe20d4942bc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:44.562844 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.562815    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-istio-envoy" pod "69ae6469-721f-4879-bd5f-132fbc422276" (UID: "69ae6469-721f-4879-bd5f-132fbc422276"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:44.563217 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.563193    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32e978e1-1585-4947-8975-fe20d4942bc5-istio-podinfo" pod "32e978e1-1585-4947-8975-fe20d4942bc5" (UID: "32e978e1-1585-4947-8975-fe20d4942bc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:44.563217 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.563238    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32e978e1-1585-4947-8975-fe20d4942bc5-kube-api-access-w6lhn" pod "32e978e1-1585-4947-8975-fe20d4942bc5" (UID: "32e978e1-1585-4947-8975-fe20d4942bc5"). InnerVolumeSpecName "kube-api-access-w6lhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:44.563579 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.563394    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ae6469-721f-4879-bd5f-132fbc422276-istio-token" pod "69ae6469-721f-4879-bd5f-132fbc422276" (UID: "69ae6469-721f-4879-bd5f-132fbc422276"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:44.563742 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.563710    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69ae6469-721f-4879-bd5f-132fbc422276-istio-podinfo" pod "69ae6469-721f-4879-bd5f-132fbc422276" (UID: "69ae6469-721f-4879-bd5f-132fbc422276"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:44.565337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.565318    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32e978e1-1585-4947-8975-fe20d4942bc5-istio-token" pod "32e978e1-1585-4947-8975-fe20d4942bc5" (UID: "32e978e1-1585-4947-8975-fe20d4942bc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:44.565477 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.565458    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-istio-envoy" pod "32e978e1-1585-4947-8975-fe20d4942bc5" (UID: "32e978e1-1585-4947-8975-fe20d4942bc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:44.702339 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.669559    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69ae6469-721f-4879-bd5f-132fbc422276-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:44.702339 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.669594    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ae6469-721f-4879-bd5f-132fbc422276-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:44.702339 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.669609    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32e978e1-1585-4947-8975-fe20d4942bc5-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:44.702339 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.669622    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6lhn\" (UniqueName: \"kubernetes.io/projected/32e978e1-1585-4947-8975-fe20d4942bc5-kube-api-access-w6lhn\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:44.702339 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.669638    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mfmw\" (UniqueName: \"kubernetes.io/projected/69ae6469-721f-4879-bd5f-132fbc422276-kube-api-access-8mfmw\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:44.702339 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.669652    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:44.702339 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.669664    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32e978e1-1585-4947-8975-fe20d4942bc5-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:44.702339 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.669677    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32e978e1-1585-4947-8975-fe20d4942bc5-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:44.702339 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:44.669689    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ae6469-721f-4879-bd5f-132fbc422276-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:45.047616 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.047583    4889 generic.go:356] "Generic (PLEG): container finished" podID="32e978e1-1585-4947-8975-fe20d4942bc5" containerID="2bde103a7436ca298adbd1ab5c82638f9ac58bd4b7aee7b0a3fd1e41fe9cef14" exitCode=0
Oct 30 22:29:45.047901 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.047657    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"
Oct 30 22:29:45.047901 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.047665    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t" event={"ID":"32e978e1-1585-4947-8975-fe20d4942bc5","Type":"ContainerDied","Data":"2bde103a7436ca298adbd1ab5c82638f9ac58bd4b7aee7b0a3fd1e41fe9cef14"}
Oct 30 22:29:45.047901 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.047715    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t" event={"ID":"32e978e1-1585-4947-8975-fe20d4942bc5","Type":"ContainerDied","Data":"d79f21e5b00a068fc9c0ff9fd2d8ea07db1d3b9debb0b8242f81ce9a328d55ba"}
Oct 30 22:29:45.047901 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.047737    4889 scope.go:122] "RemoveContainer" containerID="2bde103a7436ca298adbd1ab5c82638f9ac58bd4b7aee7b0a3fd1e41fe9cef14"
Oct 30 22:29:45.050189 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.050166    4889 generic.go:356] "Generic (PLEG): container finished" podID="69ae6469-721f-4879-bd5f-132fbc422276" containerID="b3ebf97f820d7239599f3a535efaddba67c88d9055561d5a53909d00eeee2aae" exitCode=0
Oct 30 22:29:45.050584 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.050214    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"
Oct 30 22:29:45.051731 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.050258    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh" event={"ID":"69ae6469-721f-4879-bd5f-132fbc422276","Type":"ContainerDied","Data":"b3ebf97f820d7239599f3a535efaddba67c88d9055561d5a53909d00eeee2aae"}
Oct 30 22:29:45.051831 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.051788    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh" event={"ID":"69ae6469-721f-4879-bd5f-132fbc422276","Type":"ContainerDied","Data":"f865c9810e467ed07a1246cd277d1c0fd1955012136d0bfc405b59b50ce48dd0"}
Oct 30 22:29:45.052916 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.052897    4889 generic.go:356] "Generic (PLEG): container finished" podID="6ad3d3d8-c760-4b47-925b-aa2c80d43883" containerID="ffae4b7cddb321b84e337d0db558a30d8b26430db974accef70a3c4ce893d01c" exitCode=0
Oct 30 22:29:45.053038 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.053016    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"
Oct 30 22:29:45.053103 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.053052    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr" event={"ID":"6ad3d3d8-c760-4b47-925b-aa2c80d43883","Type":"ContainerDied","Data":"ffae4b7cddb321b84e337d0db558a30d8b26430db974accef70a3c4ce893d01c"}
Oct 30 22:29:45.053103 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.053095    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr" event={"ID":"6ad3d3d8-c760-4b47-925b-aa2c80d43883","Type":"ContainerDied","Data":"cd1a45e2d9c411f98469c0d67b1bd23bc6a8e68891e9f802fa3cfcd06ae90022"}
Oct 30 22:29:45.067657 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.067204    4889 scope.go:122] "RemoveContainer" containerID="c7bcd4f15b101168d04b0b906613054c5076a4441a0a6202a6bdec339694c1f1"
Oct 30 22:29:45.182715 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.182203    4889 scope.go:122] "RemoveContainer" containerID="2bde103a7436ca298adbd1ab5c82638f9ac58bd4b7aee7b0a3fd1e41fe9cef14"
Oct 30 22:29:45.241748 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:29:45.241710    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bde103a7436ca298adbd1ab5c82638f9ac58bd4b7aee7b0a3fd1e41fe9cef14\": container with ID starting with 2bde103a7436ca298adbd1ab5c82638f9ac58bd4b7aee7b0a3fd1e41fe9cef14 not found: ID does not exist" containerID="2bde103a7436ca298adbd1ab5c82638f9ac58bd4b7aee7b0a3fd1e41fe9cef14"
Oct 30 22:29:45.242697 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.242662    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bde103a7436ca298adbd1ab5c82638f9ac58bd4b7aee7b0a3fd1e41fe9cef14"} err="failed to get container status \"2bde103a7436ca298adbd1ab5c82638f9ac58bd4b7aee7b0a3fd1e41fe9cef14\": rpc error: code = NotFound desc = could not find container \"2bde103a7436ca298adbd1ab5c82638f9ac58bd4b7aee7b0a3fd1e41fe9cef14\": container with ID starting with 2bde103a7436ca298adbd1ab5c82638f9ac58bd4b7aee7b0a3fd1e41fe9cef14 not found: ID does not exist"
Oct 30 22:29:45.242882 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.242869    4889 scope.go:122] "RemoveContainer" containerID="c7bcd4f15b101168d04b0b906613054c5076a4441a0a6202a6bdec339694c1f1"
Oct 30 22:29:45.246637 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:29:45.246601    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7bcd4f15b101168d04b0b906613054c5076a4441a0a6202a6bdec339694c1f1\": container with ID starting with c7bcd4f15b101168d04b0b906613054c5076a4441a0a6202a6bdec339694c1f1 not found: ID does not exist" containerID="c7bcd4f15b101168d04b0b906613054c5076a4441a0a6202a6bdec339694c1f1"
Oct 30 22:29:45.246833 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.246806    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7bcd4f15b101168d04b0b906613054c5076a4441a0a6202a6bdec339694c1f1"} err="failed to get container status \"c7bcd4f15b101168d04b0b906613054c5076a4441a0a6202a6bdec339694c1f1\": rpc error: code = NotFound desc = could not find container \"c7bcd4f15b101168d04b0b906613054c5076a4441a0a6202a6bdec339694c1f1\": container with ID starting with c7bcd4f15b101168d04b0b906613054c5076a4441a0a6202a6bdec339694c1f1 not found: ID does not exist"
Oct 30 22:29:45.246968 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.246953    4889 scope.go:122] "RemoveContainer" containerID="b3ebf97f820d7239599f3a535efaddba67c88d9055561d5a53909d00eeee2aae"
Oct 30 22:29:45.248899 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.248882    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"]
Oct 30 22:29:45.311313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.298217    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-95kvh"]
Oct 30 22:29:45.369249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.355583    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"]
Oct 30 22:29:45.385044 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.377426    4889 scope.go:122] "RemoveContainer" containerID="b3ebf97f820d7239599f3a535efaddba67c88d9055561d5a53909d00eeee2aae"
Oct 30 22:29:45.385044 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:29:45.378723    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3ebf97f820d7239599f3a535efaddba67c88d9055561d5a53909d00eeee2aae\": container with ID starting with b3ebf97f820d7239599f3a535efaddba67c88d9055561d5a53909d00eeee2aae not found: ID does not exist" containerID="b3ebf97f820d7239599f3a535efaddba67c88d9055561d5a53909d00eeee2aae"
Oct 30 22:29:45.385044 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.378759    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3ebf97f820d7239599f3a535efaddba67c88d9055561d5a53909d00eeee2aae"} err="failed to get container status \"b3ebf97f820d7239599f3a535efaddba67c88d9055561d5a53909d00eeee2aae\": rpc error: code = NotFound desc = could not find container \"b3ebf97f820d7239599f3a535efaddba67c88d9055561d5a53909d00eeee2aae\": container with ID starting with b3ebf97f820d7239599f3a535efaddba67c88d9055561d5a53909d00eeee2aae not found: ID does not exist"
Oct 30 22:29:45.385044 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.378785    4889 scope.go:122] "RemoveContainer" containerID="ffae4b7cddb321b84e337d0db558a30d8b26430db974accef70a3c4ce893d01c"
Oct 30 22:29:45.409254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.403672    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-w2q4t"]
Oct 30 22:29:45.456248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.450471    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"]
Oct 30 22:29:45.491044 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.474443    4889 scope.go:122] "RemoveContainer" containerID="ffae4b7cddb321b84e337d0db558a30d8b26430db974accef70a3c4ce893d01c"
Oct 30 22:29:45.491044 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:29:45.475095    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffae4b7cddb321b84e337d0db558a30d8b26430db974accef70a3c4ce893d01c\": container with ID starting with ffae4b7cddb321b84e337d0db558a30d8b26430db974accef70a3c4ce893d01c not found: ID does not exist" containerID="ffae4b7cddb321b84e337d0db558a30d8b26430db974accef70a3c4ce893d01c"
Oct 30 22:29:45.491044 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.475130    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffae4b7cddb321b84e337d0db558a30d8b26430db974accef70a3c4ce893d01c"} err="failed to get container status \"ffae4b7cddb321b84e337d0db558a30d8b26430db974accef70a3c4ce893d01c\": rpc error: code = NotFound desc = could not find container \"ffae4b7cddb321b84e337d0db558a30d8b26430db974accef70a3c4ce893d01c\": container with ID starting with ffae4b7cddb321b84e337d0db558a30d8b26430db974accef70a3c4ce893d01c not found: ID does not exist"
Oct 30 22:29:45.501155 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.501127    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-zr6tr"]
Oct 30 22:29:45.523657 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.523623    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx"]
Oct 30 22:29:45.525093 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.525065    4889 cpu_manager.go:414] "containerMap: removing container" podUID="32e978e1-1585-4947-8975-fe20d4942bc5" containerName="istio-proxy"
Oct 30 22:29:45.525284 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.525268    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32e978e1-1585-4947-8975-fe20d4942bc5" containerName="istio-proxy"
Oct 30 22:29:45.525474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.525460    4889 cpu_manager.go:414] "containerMap: removing container" podUID="6ad3d3d8-c760-4b47-925b-aa2c80d43883" containerName="istio-proxy"
Oct 30 22:29:45.525578 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.525566    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ad3d3d8-c760-4b47-925b-aa2c80d43883" containerName="istio-proxy"
Oct 30 22:29:45.525678 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.525668    4889 cpu_manager.go:414] "containerMap: removing container" podUID="69ae6469-721f-4879-bd5f-132fbc422276" containerName="istio-proxy"
Oct 30 22:29:45.525764 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.525752    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69ae6469-721f-4879-bd5f-132fbc422276" containerName="istio-proxy"
Oct 30 22:29:45.526085 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.526067    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="32e978e1-1585-4947-8975-fe20d4942bc5" containerName="istio-proxy"
Oct 30 22:29:45.526961 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.526943    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ad3d3d8-c760-4b47-925b-aa2c80d43883" containerName="istio-proxy"
Oct 30 22:29:45.527133 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.527119    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="69ae6469-721f-4879-bd5f-132fbc422276" containerName="istio-proxy"
Oct 30 22:29:45.527829 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.527812    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="32e978e1-1585-4947-8975-fe20d4942bc5" containerName="istio-proxy"
Oct 30 22:29:45.535255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.534410    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx"
Oct 30 22:29:45.550908 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.550875    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx"]
Oct 30 22:29:45.565352 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.563455    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-370ec2-serving-cert\""
Oct 30 22:29:45.643268 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.627917    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-maas-api-tls\") pod \"maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx\" (UID: \"d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx"
Oct 30 22:29:45.643268 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.628107    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-openshift-service-ca\") pod \"maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx\" (UID: \"d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx"
Oct 30 22:29:45.643268 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.628217    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-metrics-certs\") pod \"maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx\" (UID: \"d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx"
Oct 30 22:29:45.643268 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.628467    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf4pt\" (UniqueName: \"kubernetes.io/projected/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-kube-api-access-bf4pt\") pod \"maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx\" (UID: \"d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx"
Oct 30 22:29:45.701297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.701101    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:45.764937 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.746401    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-openshift-service-ca\") pod \"maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx\" (UID: \"d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx"
Oct 30 22:29:45.764937 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.746612    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-metrics-certs\") pod \"maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx\" (UID: \"d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx"
Oct 30 22:29:45.764937 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.746822    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf4pt\" (UniqueName: \"kubernetes.io/projected/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-kube-api-access-bf4pt\") pod \"maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx\" (UID: \"d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx"
Oct 30 22:29:45.764937 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.746907    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-maas-api-tls\") pod \"maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx\" (UID: \"d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx"
Oct 30 22:29:45.764937 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:45.748107    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-openshift-service-ca\") pod \"maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx\" (UID: \"d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx"
Oct 30 22:29:45.853504 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:29:45.785477    4889 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-serving-cert: secret "maas-api-e2e-api-a-370ec2-serving-cert" not found
Oct 30 22:29:45.853504 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:29:45.785577    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-maas-api-tls podName:d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35 nodeName:}" failed. No retries permitted until 2026-10-30 22:29:46.285553728 +0000 UTC m=+1371.641920034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-maas-api-tls") pod "maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx" (UID: "d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35") : secret "maas-api-e2e-api-a-370ec2-serving-cert" not found
Oct 30 22:29:46.091585 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:46.091436    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf4pt\" (UniqueName: \"kubernetes.io/projected/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-kube-api-access-bf4pt\") pod \"maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx\" (UID: \"d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx"
Oct 30 22:29:46.097647 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:46.097617    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-metrics-certs\") pod \"maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx\" (UID: \"d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx"
Oct 30 22:29:46.144406 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:46.144172    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:46.349304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:46.349268    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:46.352278 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:46.352248    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-maas-api-tls\") pod \"maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx\" (UID: \"d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx"
Oct 30 22:29:46.377173 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:46.377119    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-maas-api-tls\") pod \"maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx\" (UID: \"d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx"
Oct 30 22:29:46.494382 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:46.494349    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx"
Oct 30 22:29:46.807721 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:46.806370    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32e978e1-1585-4947-8975-fe20d4942bc5" path="/var/lib/kubelet/pods/32e978e1-1585-4947-8975-fe20d4942bc5/volumes"
Oct 30 22:29:46.807721 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:46.806880    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69ae6469-721f-4879-bd5f-132fbc422276" path="/var/lib/kubelet/pods/69ae6469-721f-4879-bd5f-132fbc422276/volumes"
Oct 30 22:29:46.807721 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:46.807319    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ad3d3d8-c760-4b47-925b-aa2c80d43883" path="/var/lib/kubelet/pods/6ad3d3d8-c760-4b47-925b-aa2c80d43883/volumes"
Oct 30 22:29:47.206887 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:47.206860    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx"]
Oct 30 22:29:47.305673 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:47.302178    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:47.306826 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:47.306787    4889 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:29:47.447251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:47.446215    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx" event={"ID":"d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35","Type":"ContainerStarted","Data":"3133c51f38a0b036e1ef6d5b715916fc075c0166781620af7a31adc31c93061d"}
Oct 30 22:29:47.524275 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:47.515911    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:48.045354 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:48.045214    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8c8cz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:29:48.045526 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:48.045363    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz" podUID="9fca6fa2-01cd-4f49-96c3-60da13e5c889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:29:48.432894 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:48.432764    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:48.474487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:48.473663    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx" event={"ID":"d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35","Type":"ContainerStarted","Data":"605853fcb101e97205a0660ede147a193138a510b44356204783b729e9b5f37a"}
Oct 30 22:29:48.474487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:48.473895    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx"
Oct 30 22:29:48.513946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:48.513872    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx" podStartSLOduration=3.074779141 podStartE2EDuration="3.513857997s" totalImagesPullingTime="439.078856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:48.510926736 +0000 UTC m=+1373.867293051" watchObservedRunningTime="2026-10-30 22:29:48.513857997 +0000 UTC m=+1373.870224309"
Oct 30 22:29:48.776308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:48.776276    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:49.097903 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.097820    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"]
Oct 30 22:29:49.098826 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.098797    4889 cpu_manager.go:414] "containerMap: removing container" podUID="32e978e1-1585-4947-8975-fe20d4942bc5" containerName="istio-proxy"
Oct 30 22:29:49.098826 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.098825    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32e978e1-1585-4947-8975-fe20d4942bc5" containerName="istio-proxy"
Oct 30 22:29:49.103057 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.103035    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.121132 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.121106    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-de31dd-openshift-default-dockercfg-q6zf7\""
Oct 30 22:29:49.139309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.139277    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"]
Oct 30 22:29:49.158644 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.158611    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-credential-socket\") pod \"e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") " pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.158823 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.158674    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-workload-socket\") pod \"e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") " pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.158823 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.158795    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp49r\" (UniqueName: \"kubernetes.io/projected/c81505a9-d460-44ad-ac11-0195c6f8bce5-kube-api-access-zp49r\") pod \"e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") " pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.158952 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.158864    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-workload-certs\") pod \"e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") " pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.158952 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.158935    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-data\") pod \"e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") " pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.159060 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.159025    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-envoy\") pod \"e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") " pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.159060 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.159049    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-podinfo\") pod \"e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") " pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.159169 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.159081    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-token\") pod \"e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") " pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.159169 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.159116    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81505a9-d460-44ad-ac11-0195c6f8bce5-istiod-ca-cert\") pod \"e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") " pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.250273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.244532    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:49.276443 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.276401    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-data\") pod \"e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") " pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.325276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.297556    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-envoy\") pod \"e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") " pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.325276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.297606    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-podinfo\") pod \"e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") " pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.325276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.297653    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-token\") pod \"e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") " pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.325276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.297695    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81505a9-d460-44ad-ac11-0195c6f8bce5-istiod-ca-cert\") pod \"e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") " pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.325276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.297845    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-credential-socket\") pod \"e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") " pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.325276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.297913    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-workload-socket\") pod \"e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") " pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.325276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.297965    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp49r\" (UniqueName: \"kubernetes.io/projected/c81505a9-d460-44ad-ac11-0195c6f8bce5-kube-api-access-zp49r\") pod \"e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") " pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.325276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.298045    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-workload-certs\") pod \"e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") " pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.325276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.294608    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-data\") pod \"e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") " pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.325276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.298852    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-workload-certs\") pod \"e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") " pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.325276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.299512    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81505a9-d460-44ad-ac11-0195c6f8bce5-istiod-ca-cert\") pod \"e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") " pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.325276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.313836    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-credential-socket\") pod \"e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") " pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.325276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.314494    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-workload-socket\") pod \"e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") " pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.325276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.315153    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-podinfo\") pod \"e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") " pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.326479 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.326446    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-envoy\") pod \"e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") " pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.361295 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.354557    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-token\") pod \"e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") " pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.361295 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.354845    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp49r\" (UniqueName: \"kubernetes.io/projected/c81505a9-d460-44ad-ac11-0195c6f8bce5-kube-api-access-zp49r\") pod \"e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") " pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.426813 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.426519    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:49.904306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.901074    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:49.940320 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:29:49.940007    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81505a9_d460_44ad_ac11_0195c6f8bce5.slice/crio-60e72d7b26ae39b4c261a1d7e6142f44b500e593e1f54bcf8175a9cd392dc3b8 WatchSource:0}: Error finding container 60e72d7b26ae39b4c261a1d7e6142f44b500e593e1f54bcf8175a9cd392dc3b8: Status 404 returned error can't find the container with id 60e72d7b26ae39b4c261a1d7e6142f44b500e593e1f54bcf8175a9cd392dc3b8
Oct 30 22:29:49.944612 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.944573    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:29:49.944742 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.944664    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:29:49.944742 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.944714    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:29:49.955403 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:49.954992    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"]
Oct 30 22:29:50.510310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:50.502972    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:50.603246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:50.598819    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq" event={"ID":"c81505a9-d460-44ad-ac11-0195c6f8bce5","Type":"ContainerStarted","Data":"e8ec72c89a7d46dcb289290ddf46b65a1985f6f3d594536b2425667a213a49bd"}
Oct 30 22:29:50.603246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:50.598888    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq" event={"ID":"c81505a9-d460-44ad-ac11-0195c6f8bce5","Type":"ContainerStarted","Data":"60e72d7b26ae39b4c261a1d7e6142f44b500e593e1f54bcf8175a9cd392dc3b8"}
Oct 30 22:29:50.969478 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:50.969392    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn"
Oct 30 22:29:51.010304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:51.010250    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq" podStartSLOduration=2.010218281 podStartE2EDuration="2.010218281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:50.66808987 +0000 UTC m=+1376.024456185" watchObservedRunningTime="2026-10-30 22:29:51.010218281 +0000 UTC m=+1376.366584594"
Oct 30 22:29:51.446702 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:51.446672    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:51.453383 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:51.453359    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:51.608487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:51.608457    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:51.609741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:51.609711    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:29:52.539247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:52.529107    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:53.209141 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:53.209107    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:53.400181 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:53.400144    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:54.345890 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:54.345850    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb"]
Oct 30 22:29:54.386260 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:54.386182    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:54.547563 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:54.547523    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:54.572499 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:54.572474    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx"
Oct 30 22:29:54.765003 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:54.764963    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:55.270241 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:55.270184    4889 scope.go:122] "RemoveContainer" containerID="cef8f79b554872c3647b8b56eda2e7064a8bd109319a539243facf040b2c124c"
Oct 30 22:29:55.288577 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:55.288557    4889 scope.go:122] "RemoveContainer" containerID="802fd5d0b6f4a8a2dd4205d5e826ce855238318dc56569db2cfac5f641f8d94d"
Oct 30 22:29:55.301907 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:55.301854    4889 scope.go:122] "RemoveContainer" containerID="d42888dc1dab5f808a283a03d24fc1c473662a6fcecf7d7ccf3d2628808135a0"
Oct 30 22:29:55.610835 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:55.610739    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:55.723429 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:55.723392    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm"]
Oct 30 22:29:55.731482 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:55.731460    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm"
Oct 30 22:29:55.744291 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:55.744260    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm"]
Oct 30 22:29:55.748642 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:55.748607    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-de31dd-dockercfg-hxb5t\""
Oct 30 22:29:55.878339 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:55.827446    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7253a4-2b57-4ac8-b8ee-11da584e0e64-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm\" (UID: \"6f7253a4-2b57-4ac8-b8ee-11da584e0e64\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm"
Oct 30 22:29:55.878339 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:55.827569    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmcdt\" (UniqueName: \"kubernetes.io/projected/6f7253a4-2b57-4ac8-b8ee-11da584e0e64-kube-api-access-mmcdt\") pod \"payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm\" (UID: \"6f7253a4-2b57-4ac8-b8ee-11da584e0e64\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm"
Oct 30 22:29:55.878339 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:55.830152    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-de31dd\""
Oct 30 22:29:55.904264 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:55.894108    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:55.985248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:55.985074    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7253a4-2b57-4ac8-b8ee-11da584e0e64-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm\" (UID: \"6f7253a4-2b57-4ac8-b8ee-11da584e0e64\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm"
Oct 30 22:29:55.985452 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:55.985270    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmcdt\" (UniqueName: \"kubernetes.io/projected/6f7253a4-2b57-4ac8-b8ee-11da584e0e64-kube-api-access-mmcdt\") pod \"payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm\" (UID: \"6f7253a4-2b57-4ac8-b8ee-11da584e0e64\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm"
Oct 30 22:29:56.000321 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:55.986805    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7253a4-2b57-4ac8-b8ee-11da584e0e64-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm\" (UID: \"6f7253a4-2b57-4ac8-b8ee-11da584e0e64\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm"
Oct 30 22:29:56.036252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:56.035714    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmcdt\" (UniqueName: \"kubernetes.io/projected/6f7253a4-2b57-4ac8-b8ee-11da584e0e64-kube-api-access-mmcdt\") pod \"payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm\" (UID: \"6f7253a4-2b57-4ac8-b8ee-11da584e0e64\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm"
Oct 30 22:29:56.077973 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:56.050032    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm"
Oct 30 22:29:56.448113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:56.448040    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w"]
Oct 30 22:29:56.454149 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:56.454129    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w"
Oct 30 22:29:56.471758 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:56.471739    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-0ff5b0-serving-cert\""
Oct 30 22:29:56.472924 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:56.472903    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w"]
Oct 30 22:29:56.485537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:56.485514    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm"]
Oct 30 22:29:56.522706 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:56.522679    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabd400f-7449-4a3b-b4ca-d969e6789f23-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w\" (UID: \"eabd400f-7449-4a3b-b4ca-d969e6789f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w"
Oct 30 22:29:56.522839 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:56.522772    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kz2c\" (UniqueName: \"kubernetes.io/projected/eabd400f-7449-4a3b-b4ca-d969e6789f23-kube-api-access-2kz2c\") pod \"maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w\" (UID: \"eabd400f-7449-4a3b-b4ca-d969e6789f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w"
Oct 30 22:29:56.522902 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:56.522838    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eabd400f-7449-4a3b-b4ca-d969e6789f23-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w\" (UID: \"eabd400f-7449-4a3b-b4ca-d969e6789f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w"
Oct 30 22:29:56.523017 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:56.522995    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabd400f-7449-4a3b-b4ca-d969e6789f23-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w\" (UID: \"eabd400f-7449-4a3b-b4ca-d969e6789f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w"
Oct 30 22:29:56.597530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:56.597494    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:56.633152 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:56.630772    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabd400f-7449-4a3b-b4ca-d969e6789f23-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w\" (UID: \"eabd400f-7449-4a3b-b4ca-d969e6789f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w"
Oct 30 22:29:56.633152 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:56.630916    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kz2c\" (UniqueName: \"kubernetes.io/projected/eabd400f-7449-4a3b-b4ca-d969e6789f23-kube-api-access-2kz2c\") pod \"maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w\" (UID: \"eabd400f-7449-4a3b-b4ca-d969e6789f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w"
Oct 30 22:29:56.633152 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:56.630953    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eabd400f-7449-4a3b-b4ca-d969e6789f23-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w\" (UID: \"eabd400f-7449-4a3b-b4ca-d969e6789f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w"
Oct 30 22:29:56.633152 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:56.631054    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabd400f-7449-4a3b-b4ca-d969e6789f23-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w\" (UID: \"eabd400f-7449-4a3b-b4ca-d969e6789f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w"
Oct 30 22:29:56.648273 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:29:56.643045    4889 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-serving-cert: secret "maas-api-e2e-shared-a-w6-0ff5b0-serving-cert" not found
Oct 30 22:29:56.648273 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:29:56.643149    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eabd400f-7449-4a3b-b4ca-d969e6789f23-maas-api-tls podName:eabd400f-7449-4a3b-b4ca-d969e6789f23 nodeName:}" failed. No retries permitted until 2026-10-30 22:29:57.14312517 +0000 UTC m=+1382.499491475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eabd400f-7449-4a3b-b4ca-d969e6789f23-maas-api-tls") pod "maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w" (UID: "eabd400f-7449-4a3b-b4ca-d969e6789f23") : secret "maas-api-e2e-shared-a-w6-0ff5b0-serving-cert" not found
Oct 30 22:29:56.648273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:56.644101    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabd400f-7449-4a3b-b4ca-d969e6789f23-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w\" (UID: \"eabd400f-7449-4a3b-b4ca-d969e6789f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w"
Oct 30 22:29:56.648273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:56.645010    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabd400f-7449-4a3b-b4ca-d969e6789f23-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w\" (UID: \"eabd400f-7449-4a3b-b4ca-d969e6789f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w"
Oct 30 22:29:56.662249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:56.660897    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kz2c\" (UniqueName: \"kubernetes.io/projected/eabd400f-7449-4a3b-b4ca-d969e6789f23-kube-api-access-2kz2c\") pod \"maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w\" (UID: \"eabd400f-7449-4a3b-b4ca-d969e6789f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w"
Oct 30 22:29:56.771645 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:56.771613    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:56.961465 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:56.961435    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:57.012260 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:57.012196    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm" event={"ID":"6f7253a4-2b57-4ac8-b8ee-11da584e0e64","Type":"ContainerStarted","Data":"d1622b8e416d1a4e0fda40b01edd3b047eb8de380a3461a7caa7d3f194f3cd2e"}
Oct 30 22:29:57.012423 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:57.012269    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm" event={"ID":"6f7253a4-2b57-4ac8-b8ee-11da584e0e64","Type":"ContainerStarted","Data":"e7dba7b91d42980b4299cf14191aacfbdbfb58d4ddb2e2ffabffed7ec02b4cdd"}
Oct 30 22:29:57.012472 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:57.012460    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm"
Oct 30 22:29:57.049559 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:57.049461    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm" podStartSLOduration=2.049447213 podStartE2EDuration="2.049447213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:57.047529453 +0000 UTC m=+1382.403895771" watchObservedRunningTime="2026-10-30 22:29:57.049447213 +0000 UTC m=+1382.405813526"
Oct 30 22:29:57.143216 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:57.143179    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eabd400f-7449-4a3b-b4ca-d969e6789f23-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w\" (UID: \"eabd400f-7449-4a3b-b4ca-d969e6789f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w"
Oct 30 22:29:57.145895 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:57.145872    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eabd400f-7449-4a3b-b4ca-d969e6789f23-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w\" (UID: \"eabd400f-7449-4a3b-b4ca-d969e6789f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w"
Oct 30 22:29:57.374045 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:57.373949    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w"
Oct 30 22:29:57.833560 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:57.833525    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w"]
Oct 30 22:29:58.035252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:58.034398    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w" event={"ID":"eabd400f-7449-4a3b-b4ca-d969e6789f23","Type":"ContainerStarted","Data":"c909453bfbd61863c755cb85c95a9d00b3ad6c0042a45f42e52e41ba5773c9c9"}
Oct 30 22:29:58.653280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:58.653197    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:59.042679 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:59.042641    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w" event={"ID":"eabd400f-7449-4a3b-b4ca-d969e6789f23","Type":"ContainerStarted","Data":"cc7edb72a5416e74269e4e05015ee87a47e7c0bac125dc3dcb2578ced15590bb"}
Oct 30 22:29:59.043162 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:59.042857    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w"
Oct 30 22:29:59.084940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:59.084879    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w" podStartSLOduration=2.700468758 podStartE2EDuration="3.084865708s" totalImagesPullingTime="384.39695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:59.082854875 +0000 UTC m=+1384.439221250" watchObservedRunningTime="2026-10-30 22:29:59.084865708 +0000 UTC m=+1384.441232055"
Oct 30 22:29:59.346755 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:59.346664    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb" podUID="77442c89-ba79-4612-85eb-cab9b6cc702c" containerName="payload-processing" containerID="cri-o://f77b786bc6abbfe05720843479c226296fdcbef7601c9e691d2a29d4eaae67aa" gracePeriod=25
Oct 30 22:29:59.645612 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:59.645586    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb"
Oct 30 22:29:59.822191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:59.822155    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77442c89-ba79-4612-85eb-cab9b6cc702c-kube-api-access-vzpc4\" (UniqueName: \"kubernetes.io/projected/77442c89-ba79-4612-85eb-cab9b6cc702c-kube-api-access-vzpc4\") pod \"77442c89-ba79-4612-85eb-cab9b6cc702c\" (UID: \"77442c89-ba79-4612-85eb-cab9b6cc702c\") "
Oct 30 22:29:59.822373 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:59.822236    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77442c89-ba79-4612-85eb-cab9b6cc702c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77442c89-ba79-4612-85eb-cab9b6cc702c-plugins-config-volume\") pod \"77442c89-ba79-4612-85eb-cab9b6cc702c\" (UID: \"77442c89-ba79-4612-85eb-cab9b6cc702c\") "
Oct 30 22:29:59.822681 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:59.822650    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77442c89-ba79-4612-85eb-cab9b6cc702c-plugins-config-volume" pod "77442c89-ba79-4612-85eb-cab9b6cc702c" (UID: "77442c89-ba79-4612-85eb-cab9b6cc702c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:59.824755 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:59.824729    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77442c89-ba79-4612-85eb-cab9b6cc702c-kube-api-access-vzpc4" pod "77442c89-ba79-4612-85eb-cab9b6cc702c" (UID: "77442c89-ba79-4612-85eb-cab9b6cc702c"). InnerVolumeSpecName "kube-api-access-vzpc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:59.924853 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:59.924781    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzpc4\" (UniqueName: \"kubernetes.io/projected/77442c89-ba79-4612-85eb-cab9b6cc702c-kube-api-access-vzpc4\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:29:59.924853 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:29:59.924804    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77442c89-ba79-4612-85eb-cab9b6cc702c-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:30:00.052166 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.052131    4889 generic.go:356] "Generic (PLEG): container finished" podID="77442c89-ba79-4612-85eb-cab9b6cc702c" containerID="f77b786bc6abbfe05720843479c226296fdcbef7601c9e691d2a29d4eaae67aa" exitCode=0
Oct 30 22:30:00.052689 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.052182    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb"
Oct 30 22:30:00.052689 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.052217    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb" event={"ID":"77442c89-ba79-4612-85eb-cab9b6cc702c","Type":"ContainerDied","Data":"f77b786bc6abbfe05720843479c226296fdcbef7601c9e691d2a29d4eaae67aa"}
Oct 30 22:30:00.052689 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.052313    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb" event={"ID":"77442c89-ba79-4612-85eb-cab9b6cc702c","Type":"ContainerDied","Data":"7c86b1700f0fa37cc970e98b2040e71ad2c29d5e2ddbb470c6c45a19aee506d4"}
Oct 30 22:30:00.052689 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.052342    4889 scope.go:122] "RemoveContainer" containerID="f77b786bc6abbfe05720843479c226296fdcbef7601c9e691d2a29d4eaae67aa"
Oct 30 22:30:00.063511 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.063493    4889 scope.go:122] "RemoveContainer" containerID="f77b786bc6abbfe05720843479c226296fdcbef7601c9e691d2a29d4eaae67aa"
Oct 30 22:30:00.063794 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:30:00.063775    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f77b786bc6abbfe05720843479c226296fdcbef7601c9e691d2a29d4eaae67aa\": container with ID starting with f77b786bc6abbfe05720843479c226296fdcbef7601c9e691d2a29d4eaae67aa not found: ID does not exist" containerID="f77b786bc6abbfe05720843479c226296fdcbef7601c9e691d2a29d4eaae67aa"
Oct 30 22:30:00.063860 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.063814    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f77b786bc6abbfe05720843479c226296fdcbef7601c9e691d2a29d4eaae67aa"} err="failed to get container status \"f77b786bc6abbfe05720843479c226296fdcbef7601c9e691d2a29d4eaae67aa\": rpc error: code = NotFound desc = could not find container \"f77b786bc6abbfe05720843479c226296fdcbef7601c9e691d2a29d4eaae67aa\": container with ID starting with f77b786bc6abbfe05720843479c226296fdcbef7601c9e691d2a29d4eaae67aa not found: ID does not exist"
Oct 30 22:30:00.103947 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.103918    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb"]
Oct 30 22:30:00.115168 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.115141    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d399fb07-56bfdc9c75-7d8qb"]
Oct 30 22:30:00.184739 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.184653    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj"]
Oct 30 22:30:00.185918 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.185891    4889 cpu_manager.go:414] "containerMap: removing container" podUID="77442c89-ba79-4612-85eb-cab9b6cc702c" containerName="payload-processing"
Oct 30 22:30:00.185918 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.185916    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77442c89-ba79-4612-85eb-cab9b6cc702c" containerName="payload-processing"
Oct 30 22:30:00.186217 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.186199    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="77442c89-ba79-4612-85eb-cab9b6cc702c" containerName="payload-processing"
Oct 30 22:30:00.192343 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.192319    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj"
Oct 30 22:30:00.198067 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.198044    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd"]
Oct 30 22:30:00.204701 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.204674    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd"
Oct 30 22:30:00.206394 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.206324    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn"]
Oct 30 22:30:00.214777 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.214751    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj"]
Oct 30 22:30:00.215245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.214863    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn"
Oct 30 22:30:00.223985 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.223958    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd"]
Oct 30 22:30:00.233946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.233917    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn"]
Oct 30 22:30:00.294849 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.294824    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:30:00.332366 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.332337    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ff76\" (UniqueName: \"kubernetes.io/projected/dfed6566-0e74-47e5-aa5f-edff545d4297-kube-api-access-6ff76\") pod \"maas-api-key-cleanup-29889990-m6ptj\" (UID: \"dfed6566-0e74-47e5-aa5f-edff545d4297\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj"
Oct 30 22:30:00.332555 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.332526    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4422\" (UniqueName: \"kubernetes.io/projected/4091a3c0-1f2f-4444-ae6b-20fc34f19e4c-kube-api-access-s4422\") pod \"maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd\" (UID: \"4091a3c0-1f2f-4444-ae6b-20fc34f19e4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd"
Oct 30 22:30:00.332659 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.332631    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zb6m\" (UniqueName: \"kubernetes.io/projected/c9674a90-e299-4582-9375-0e6a9bed0e4a-kube-api-access-7zb6m\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn\" (UID: \"c9674a90-e299-4582-9375-0e6a9bed0e4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn"
Oct 30 22:30:00.332753 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.332708    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4091a3c0-1f2f-4444-ae6b-20fc34f19e4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd\" (UID: \"4091a3c0-1f2f-4444-ae6b-20fc34f19e4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd"
Oct 30 22:30:00.332797 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.332762    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfed6566-0e74-47e5-aa5f-edff545d4297-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-m6ptj\" (UID: \"dfed6566-0e74-47e5-aa5f-edff545d4297\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj"
Oct 30 22:30:00.332864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.332794    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9674a90-e299-4582-9375-0e6a9bed0e4a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn\" (UID: \"c9674a90-e299-4582-9375-0e6a9bed0e4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn"
Oct 30 22:30:00.434397 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.434360    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ff76\" (UniqueName: \"kubernetes.io/projected/dfed6566-0e74-47e5-aa5f-edff545d4297-kube-api-access-6ff76\") pod \"maas-api-key-cleanup-29889990-m6ptj\" (UID: \"dfed6566-0e74-47e5-aa5f-edff545d4297\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj"
Oct 30 22:30:00.434597 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.434543    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4422\" (UniqueName: \"kubernetes.io/projected/4091a3c0-1f2f-4444-ae6b-20fc34f19e4c-kube-api-access-s4422\") pod \"maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd\" (UID: \"4091a3c0-1f2f-4444-ae6b-20fc34f19e4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd"
Oct 30 22:30:00.434667 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.434598    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zb6m\" (UniqueName: \"kubernetes.io/projected/c9674a90-e299-4582-9375-0e6a9bed0e4a-kube-api-access-7zb6m\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn\" (UID: \"c9674a90-e299-4582-9375-0e6a9bed0e4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn"
Oct 30 22:30:00.434667 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.434657    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4091a3c0-1f2f-4444-ae6b-20fc34f19e4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd\" (UID: \"4091a3c0-1f2f-4444-ae6b-20fc34f19e4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd"
Oct 30 22:30:00.434804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.434685    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfed6566-0e74-47e5-aa5f-edff545d4297-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-m6ptj\" (UID: \"dfed6566-0e74-47e5-aa5f-edff545d4297\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj"
Oct 30 22:30:00.434804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.434727    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9674a90-e299-4582-9375-0e6a9bed0e4a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn\" (UID: \"c9674a90-e299-4582-9375-0e6a9bed0e4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn"
Oct 30 22:30:00.435580 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.435559    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9674a90-e299-4582-9375-0e6a9bed0e4a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn\" (UID: \"c9674a90-e299-4582-9375-0e6a9bed0e4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn"
Oct 30 22:30:00.435675 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.435589    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfed6566-0e74-47e5-aa5f-edff545d4297-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-m6ptj\" (UID: \"dfed6566-0e74-47e5-aa5f-edff545d4297\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj"
Oct 30 22:30:00.435675 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.435602    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4091a3c0-1f2f-4444-ae6b-20fc34f19e4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd\" (UID: \"4091a3c0-1f2f-4444-ae6b-20fc34f19e4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd"
Oct 30 22:30:00.453353 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.453332    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4422\" (UniqueName: \"kubernetes.io/projected/4091a3c0-1f2f-4444-ae6b-20fc34f19e4c-kube-api-access-s4422\") pod \"maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd\" (UID: \"4091a3c0-1f2f-4444-ae6b-20fc34f19e4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd"
Oct 30 22:30:00.453444 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.453416    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ff76\" (UniqueName: \"kubernetes.io/projected/dfed6566-0e74-47e5-aa5f-edff545d4297-kube-api-access-6ff76\") pod \"maas-api-key-cleanup-29889990-m6ptj\" (UID: \"dfed6566-0e74-47e5-aa5f-edff545d4297\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj"
Oct 30 22:30:00.453534 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.453513    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zb6m\" (UniqueName: \"kubernetes.io/projected/c9674a90-e299-4582-9375-0e6a9bed0e4a-kube-api-access-7zb6m\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn\" (UID: \"c9674a90-e299-4582-9375-0e6a9bed0e4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn"
Oct 30 22:30:00.542627 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.538832    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:00.542627 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.541478    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj"
Oct 30 22:30:00.542627 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.541847    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd"
Oct 30 22:30:00.576090 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.575424    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn"
Oct 30 22:30:00.782157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.782127    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77442c89-ba79-4612-85eb-cab9b6cc702c" path="/var/lib/kubelet/pods/77442c89-ba79-4612-85eb-cab9b6cc702c/volumes"
Oct 30 22:30:00.925577 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:30:00.925546    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfed6566_0e74_47e5_aa5f_edff545d4297.slice/crio-6f2a48dae6f1999babf82bb7397e82fcb87deae34b7c55cdf86298f57d204f5c WatchSource:0}: Error finding container 6f2a48dae6f1999babf82bb7397e82fcb87deae34b7c55cdf86298f57d204f5c: Status 404 returned error can't find the container with id 6f2a48dae6f1999babf82bb7397e82fcb87deae34b7c55cdf86298f57d204f5c
Oct 30 22:30:00.925675 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.925589    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj"]
Oct 30 22:30:00.958282 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:30:00.958255    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4091a3c0_1f2f_4444_ae6b_20fc34f19e4c.slice/crio-38b983d331ff4cab2b10fd1beb9f8c0170f2d3d583fecdb9f70bb6df7adc3347 WatchSource:0}: Error finding container 38b983d331ff4cab2b10fd1beb9f8c0170f2d3d583fecdb9f70bb6df7adc3347: Status 404 returned error can't find the container with id 38b983d331ff4cab2b10fd1beb9f8c0170f2d3d583fecdb9f70bb6df7adc3347
Oct 30 22:30:00.958678 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.958656    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd"]
Oct 30 22:30:00.965010 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:00.964979    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn"]
Oct 30 22:30:00.966558 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:30:00.966508    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9674a90_e299_4582_9375_0e6a9bed0e4a.slice/crio-b60ec17ff9a9982823459e2a1014949e63cf45cf01f7e05f7fa26ebcbc6947c6 WatchSource:0}: Error finding container b60ec17ff9a9982823459e2a1014949e63cf45cf01f7e05f7fa26ebcbc6947c6: Status 404 returned error can't find the container with id b60ec17ff9a9982823459e2a1014949e63cf45cf01f7e05f7fa26ebcbc6947c6
Oct 30 22:30:01.061790 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:01.061750    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn" event={"ID":"c9674a90-e299-4582-9375-0e6a9bed0e4a","Type":"ContainerStarted","Data":"b60ec17ff9a9982823459e2a1014949e63cf45cf01f7e05f7fa26ebcbc6947c6"}
Oct 30 22:30:01.063157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:01.063130    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj" event={"ID":"dfed6566-0e74-47e5-aa5f-edff545d4297","Type":"ContainerStarted","Data":"6f2a48dae6f1999babf82bb7397e82fcb87deae34b7c55cdf86298f57d204f5c"}
Oct 30 22:30:01.064499 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:01.064474    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd" event={"ID":"4091a3c0-1f2f-4444-ae6b-20fc34f19e4c","Type":"ContainerStarted","Data":"38b983d331ff4cab2b10fd1beb9f8c0170f2d3d583fecdb9f70bb6df7adc3347"}
Oct 30 22:30:02.223430 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:02.223395    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft"]
Oct 30 22:30:02.229518 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:02.229493    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft"
Oct 30 22:30:02.251748 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:02.250792    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft"]
Oct 30 22:30:02.252181 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:02.252129    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f443d6-serving-cert\""
Oct 30 22:30:02.267434 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:02.267406    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09676531-ce1f-4950-b903-730935715f8a-maas-api-tls\") pod \"maas-api-e2e-api-a-f443d6-7c96986b84-52sft\" (UID: \"09676531-ce1f-4950-b903-730935715f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft"
Oct 30 22:30:02.267577 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:02.267448    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09676531-ce1f-4950-b903-730935715f8a-openshift-service-ca\") pod \"maas-api-e2e-api-a-f443d6-7c96986b84-52sft\" (UID: \"09676531-ce1f-4950-b903-730935715f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft"
Oct 30 22:30:02.267577 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:02.267561    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-569ql\" (UniqueName: \"kubernetes.io/projected/09676531-ce1f-4950-b903-730935715f8a-kube-api-access-569ql\") pod \"maas-api-e2e-api-a-f443d6-7c96986b84-52sft\" (UID: \"09676531-ce1f-4950-b903-730935715f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft"
Oct 30 22:30:02.267703 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:02.267690    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09676531-ce1f-4950-b903-730935715f8a-metrics-certs\") pod \"maas-api-e2e-api-a-f443d6-7c96986b84-52sft\" (UID: \"09676531-ce1f-4950-b903-730935715f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft"
Oct 30 22:30:02.362236 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:02.362175    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:02.369129 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:02.369104    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-569ql\" (UniqueName: \"kubernetes.io/projected/09676531-ce1f-4950-b903-730935715f8a-kube-api-access-569ql\") pod \"maas-api-e2e-api-a-f443d6-7c96986b84-52sft\" (UID: \"09676531-ce1f-4950-b903-730935715f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft"
Oct 30 22:30:02.369525 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:02.369502    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09676531-ce1f-4950-b903-730935715f8a-metrics-certs\") pod \"maas-api-e2e-api-a-f443d6-7c96986b84-52sft\" (UID: \"09676531-ce1f-4950-b903-730935715f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft"
Oct 30 22:30:02.369704 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:02.369687    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09676531-ce1f-4950-b903-730935715f8a-maas-api-tls\") pod \"maas-api-e2e-api-a-f443d6-7c96986b84-52sft\" (UID: \"09676531-ce1f-4950-b903-730935715f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft"
Oct 30 22:30:02.369832 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:02.369816    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09676531-ce1f-4950-b903-730935715f8a-openshift-service-ca\") pod \"maas-api-e2e-api-a-f443d6-7c96986b84-52sft\" (UID: \"09676531-ce1f-4950-b903-730935715f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft"
Oct 30 22:30:02.370825 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:02.370800    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09676531-ce1f-4950-b903-730935715f8a-openshift-service-ca\") pod \"maas-api-e2e-api-a-f443d6-7c96986b84-52sft\" (UID: \"09676531-ce1f-4950-b903-730935715f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft"
Oct 30 22:30:02.373087 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:02.373063    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09676531-ce1f-4950-b903-730935715f8a-metrics-certs\") pod \"maas-api-e2e-api-a-f443d6-7c96986b84-52sft\" (UID: \"09676531-ce1f-4950-b903-730935715f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft"
Oct 30 22:30:02.375105 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:02.375078    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09676531-ce1f-4950-b903-730935715f8a-maas-api-tls\") pod \"maas-api-e2e-api-a-f443d6-7c96986b84-52sft\" (UID: \"09676531-ce1f-4950-b903-730935715f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft"
Oct 30 22:30:02.401084 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:02.397305    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-569ql\" (UniqueName: \"kubernetes.io/projected/09676531-ce1f-4950-b903-730935715f8a-kube-api-access-569ql\") pod \"maas-api-e2e-api-a-f443d6-7c96986b84-52sft\" (UID: \"09676531-ce1f-4950-b903-730935715f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft"
Oct 30 22:30:02.571248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:02.570620    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft"
Oct 30 22:30:02.616343 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:02.615717    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:02.960556 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:02.960510    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft"]
Oct 30 22:30:02.960721 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:30:02.960633    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09676531_ce1f_4950_b903_730935715f8a.slice/crio-a02cfc44b6d11cc2fd4f09af2c4b5fc021809ffa450acac93a246e17807c4e66 WatchSource:0}: Error finding container a02cfc44b6d11cc2fd4f09af2c4b5fc021809ffa450acac93a246e17807c4e66: Status 404 returned error can't find the container with id a02cfc44b6d11cc2fd4f09af2c4b5fc021809ffa450acac93a246e17807c4e66
Oct 30 22:30:03.088908 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:03.088874    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft" event={"ID":"09676531-ce1f-4950-b903-730935715f8a","Type":"ContainerStarted","Data":"a02cfc44b6d11cc2fd4f09af2c4b5fc021809ffa450acac93a246e17807c4e66"}
Oct 30 22:30:04.103267 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:04.103198    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft" event={"ID":"09676531-ce1f-4950-b903-730935715f8a","Type":"ContainerStarted","Data":"9cf2b97303353fbcec60dd07aa09c4ee696518ad02c7b39256cf49d7dbeac310"}
Oct 30 22:30:04.103761 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:04.103305    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft"
Oct 30 22:30:04.149410 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:04.149344    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft" podStartSLOduration=1.817550478 podStartE2EDuration="2.149324609s" totalImagesPullingTime="331.774131ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:04.141440605 +0000 UTC m=+1389.497807000" watchObservedRunningTime="2026-10-30 22:30:04.149324609 +0000 UTC m=+1389.505690904"
Oct 30 22:30:04.426425 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:04.426338    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:05.059814 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:05.059725    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w"
Oct 30 22:30:06.119451 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:06.119406    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9"]
Oct 30 22:30:06.124290 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:06.124268    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9"
Oct 30 22:30:06.127592 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:06.127562    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn" event={"ID":"c9674a90-e299-4582-9375-0e6a9bed0e4a","Type":"ContainerStarted","Data":"3acceff5f01c86546721ef910654fa3ecb67f73086e54014f348dfaa71b3b141"}
Oct 30 22:30:06.129444 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:06.129420    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj" event={"ID":"dfed6566-0e74-47e5-aa5f-edff545d4297","Type":"ContainerStarted","Data":"ab0f7d7147aac7b2ac23336abac913a9dd2073c91dd0de13489b960653b729e1"}
Oct 30 22:30:06.131190 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:06.131166    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd" event={"ID":"4091a3c0-1f2f-4444-ae6b-20fc34f19e4c","Type":"ContainerStarted","Data":"d0d1b81ce2bb49d412b56fded6f54a52d7a7927a75a4ad28bbeee10fad712d6d"}
Oct 30 22:30:06.161559 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:06.149933    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2949c143-serving-cert\""
Oct 30 22:30:06.161559 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:06.159675    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9"]
Oct 30 22:30:06.220808 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:06.220752    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd" podStartSLOduration=1.87980799 podStartE2EDuration="6.220734683s" totalImagesPullingTime="4.340926693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:06.211686628 +0000 UTC m=+1391.568052942" watchObservedRunningTime="2026-10-30 22:30:06.220734683 +0000 UTC m=+1391.577101005"
Oct 30 22:30:06.246897 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:06.246862    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a7d79a6-3485-49c6-98f9-e83552ec94e7-maas-api-tls\") pod \"maas-api-e2e-mt-2949c143-5d858999bf-f5dz9\" (UID: \"3a7d79a6-3485-49c6-98f9-e83552ec94e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9"
Oct 30 22:30:06.246897 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:06.246859    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn" podStartSLOduration=1.790893234 podStartE2EDuration="6.246842534s" totalImagesPullingTime="4.4559493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:06.24609002 +0000 UTC m=+1391.602456333" watchObservedRunningTime="2026-10-30 22:30:06.246842534 +0000 UTC m=+1391.603208850"
Oct 30 22:30:06.247135 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:06.247052    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfldk\" (UniqueName: \"kubernetes.io/projected/3a7d79a6-3485-49c6-98f9-e83552ec94e7-kube-api-access-jfldk\") pod \"maas-api-e2e-mt-2949c143-5d858999bf-f5dz9\" (UID: \"3a7d79a6-3485-49c6-98f9-e83552ec94e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9"
Oct 30 22:30:06.247331 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:06.247307    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a7d79a6-3485-49c6-98f9-e83552ec94e7-metrics-certs\") pod \"maas-api-e2e-mt-2949c143-5d858999bf-f5dz9\" (UID: \"3a7d79a6-3485-49c6-98f9-e83552ec94e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9"
Oct 30 22:30:06.247331 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:06.247440    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7d79a6-3485-49c6-98f9-e83552ec94e7-openshift-service-ca\") pod \"maas-api-e2e-mt-2949c143-5d858999bf-f5dz9\" (UID: \"3a7d79a6-3485-49c6-98f9-e83552ec94e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9"
Oct 30 22:30:06.284846 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:06.284750    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj" podStartSLOduration=1.905544374 podStartE2EDuration="6.284736847s" totalImagesPullingTime="4.379192473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:06.281181759 +0000 UTC m=+1391.637548072" watchObservedRunningTime="2026-10-30 22:30:06.284736847 +0000 UTC m=+1391.641103179"
Oct 30 22:30:06.358257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:06.355713    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:06.368255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:06.366689    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfldk\" (UniqueName: \"kubernetes.io/projected/3a7d79a6-3485-49c6-98f9-e83552ec94e7-kube-api-access-jfldk\") pod \"maas-api-e2e-mt-2949c143-5d858999bf-f5dz9\" (UID: \"3a7d79a6-3485-49c6-98f9-e83552ec94e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9"
Oct 30 22:30:06.368255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:06.366845    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a7d79a6-3485-49c6-98f9-e83552ec94e7-metrics-certs\") pod \"maas-api-e2e-mt-2949c143-5d858999bf-f5dz9\" (UID: \"3a7d79a6-3485-49c6-98f9-e83552ec94e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9"
Oct 30 22:30:06.368255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:06.366957    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7d79a6-3485-49c6-98f9-e83552ec94e7-openshift-service-ca\") pod \"maas-api-e2e-mt-2949c143-5d858999bf-f5dz9\" (UID: \"3a7d79a6-3485-49c6-98f9-e83552ec94e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9"
Oct 30 22:30:06.368255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:06.367148    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a7d79a6-3485-49c6-98f9-e83552ec94e7-maas-api-tls\") pod \"maas-api-e2e-mt-2949c143-5d858999bf-f5dz9\" (UID: \"3a7d79a6-3485-49c6-98f9-e83552ec94e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9"
Oct 30 22:30:06.392251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:06.387637    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7d79a6-3485-49c6-98f9-e83552ec94e7-openshift-service-ca\") pod \"maas-api-e2e-mt-2949c143-5d858999bf-f5dz9\" (UID: \"3a7d79a6-3485-49c6-98f9-e83552ec94e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9"
Oct 30 22:30:06.395306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:06.393543    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a7d79a6-3485-49c6-98f9-e83552ec94e7-maas-api-tls\") pod \"maas-api-e2e-mt-2949c143-5d858999bf-f5dz9\" (UID: \"3a7d79a6-3485-49c6-98f9-e83552ec94e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9"
Oct 30 22:30:06.409263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:06.405694    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a7d79a6-3485-49c6-98f9-e83552ec94e7-metrics-certs\") pod \"maas-api-e2e-mt-2949c143-5d858999bf-f5dz9\" (UID: \"3a7d79a6-3485-49c6-98f9-e83552ec94e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9"
Oct 30 22:30:06.420600 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:06.417132    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfldk\" (UniqueName: \"kubernetes.io/projected/3a7d79a6-3485-49c6-98f9-e83552ec94e7-kube-api-access-jfldk\") pod \"maas-api-e2e-mt-2949c143-5d858999bf-f5dz9\" (UID: \"3a7d79a6-3485-49c6-98f9-e83552ec94e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9"
Oct 30 22:30:06.484027 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:06.444564    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9"
Oct 30 22:30:06.524336 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:06.524300    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:06.800673 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:30:06.800608    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a7d79a6_3485_49c6_98f9_e83552ec94e7.slice/crio-944417d64afc46c750d26701b22390066438a828bb0f1f0f1ed5eea86583f307 WatchSource:0}: Error finding container 944417d64afc46c750d26701b22390066438a828bb0f1f0f1ed5eea86583f307: Status 404 returned error can't find the container with id 944417d64afc46c750d26701b22390066438a828bb0f1f0f1ed5eea86583f307
Oct 30 22:30:06.800673 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:06.800643    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9"]
Oct 30 22:30:07.141421 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:07.141388    4889 generic.go:356] "Generic (PLEG): container finished" podID="c9674a90-e299-4582-9375-0e6a9bed0e4a" containerID="3acceff5f01c86546721ef910654fa3ecb67f73086e54014f348dfaa71b3b141" exitCode=60
Oct 30 22:30:07.141839 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:07.141481    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn" event={"ID":"c9674a90-e299-4582-9375-0e6a9bed0e4a","Type":"ContainerDied","Data":"3acceff5f01c86546721ef910654fa3ecb67f73086e54014f348dfaa71b3b141"}
Oct 30 22:30:07.141915 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:07.141898    4889 scope.go:122] "RemoveContainer" containerID="3acceff5f01c86546721ef910654fa3ecb67f73086e54014f348dfaa71b3b141"
Oct 30 22:30:07.143020 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:07.142990    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9" event={"ID":"3a7d79a6-3485-49c6-98f9-e83552ec94e7","Type":"ContainerStarted","Data":"944417d64afc46c750d26701b22390066438a828bb0f1f0f1ed5eea86583f307"}
Oct 30 22:30:07.144662 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:07.144643    4889 generic.go:356] "Generic (PLEG): container finished" podID="dfed6566-0e74-47e5-aa5f-edff545d4297" containerID="ab0f7d7147aac7b2ac23336abac913a9dd2073c91dd0de13489b960653b729e1" exitCode=60
Oct 30 22:30:07.144740 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:07.144719    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj" event={"ID":"dfed6566-0e74-47e5-aa5f-edff545d4297","Type":"ContainerDied","Data":"ab0f7d7147aac7b2ac23336abac913a9dd2073c91dd0de13489b960653b729e1"}
Oct 30 22:30:07.145004 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:07.144986    4889 scope.go:122] "RemoveContainer" containerID="ab0f7d7147aac7b2ac23336abac913a9dd2073c91dd0de13489b960653b729e1"
Oct 30 22:30:07.146470 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:07.146448    4889 generic.go:356] "Generic (PLEG): container finished" podID="4091a3c0-1f2f-4444-ae6b-20fc34f19e4c" containerID="d0d1b81ce2bb49d412b56fded6f54a52d7a7927a75a4ad28bbeee10fad712d6d" exitCode=60
Oct 30 22:30:07.146607 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:07.146586    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd" event={"ID":"4091a3c0-1f2f-4444-ae6b-20fc34f19e4c","Type":"ContainerDied","Data":"d0d1b81ce2bb49d412b56fded6f54a52d7a7927a75a4ad28bbeee10fad712d6d"}
Oct 30 22:30:07.146968 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:07.146861    4889 scope.go:122] "RemoveContainer" containerID="d0d1b81ce2bb49d412b56fded6f54a52d7a7927a75a4ad28bbeee10fad712d6d"
Oct 30 22:30:08.038243 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:08.038202    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm"
Oct 30 22:30:08.155836 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:08.155805    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:08.156388 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:08.156366    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd" event={"ID":"4091a3c0-1f2f-4444-ae6b-20fc34f19e4c","Type":"ContainerStarted","Data":"3cc54442170da199bb84e6939d5508068622770beb5ba871054f1d684aff4a17"}
Oct 30 22:30:08.159028 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:08.158998    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn" event={"ID":"c9674a90-e299-4582-9375-0e6a9bed0e4a","Type":"ContainerStarted","Data":"eedac7c8f0c375595af10c59c17407f70acd97675f73edc2f6133a4f3e04a939"}
Oct 30 22:30:08.160933 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:08.160904    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9" event={"ID":"3a7d79a6-3485-49c6-98f9-e83552ec94e7","Type":"ContainerStarted","Data":"377d6daef8f8d3a183db1d630419ff74c53e4ba180bc0833a469ec75d02e982c"}
Oct 30 22:30:08.161045 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:08.160961    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9"
Oct 30 22:30:08.162969 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:08.162946    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj" event={"ID":"dfed6566-0e74-47e5-aa5f-edff545d4297","Type":"ContainerStarted","Data":"38a4d184c6d74bc75a7d33ce4b1525e82351c6520ceed4126bc5dd9ecf934768"}
Oct 30 22:30:08.290324 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:08.284466    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9" podStartSLOduration=1.9948638760000001 podStartE2EDuration="2.28444438s" totalImagesPullingTime="289.580504ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:08.277913626 +0000 UTC m=+1393.634279940" watchObservedRunningTime="2026-10-30 22:30:08.28444438 +0000 UTC m=+1393.640810686"
Oct 30 22:30:09.174967 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:09.174930    4889 generic.go:356] "Generic (PLEG): container finished" podID="dfed6566-0e74-47e5-aa5f-edff545d4297" containerID="38a4d184c6d74bc75a7d33ce4b1525e82351c6520ceed4126bc5dd9ecf934768" exitCode=60
Oct 30 22:30:09.175487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:09.175027    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj" event={"ID":"dfed6566-0e74-47e5-aa5f-edff545d4297","Type":"ContainerDied","Data":"38a4d184c6d74bc75a7d33ce4b1525e82351c6520ceed4126bc5dd9ecf934768"}
Oct 30 22:30:09.175487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:09.175103    4889 scope.go:122] "RemoveContainer" containerID="ab0f7d7147aac7b2ac23336abac913a9dd2073c91dd0de13489b960653b729e1"
Oct 30 22:30:09.175487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:09.175368    4889 scope.go:122] "RemoveContainer" containerID="38a4d184c6d74bc75a7d33ce4b1525e82351c6520ceed4126bc5dd9ecf934768"
Oct 30 22:30:09.175870 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:30:09.175840    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-m6ptj_odh-ai-gateway-infra(dfed6566-0e74-47e5-aa5f-edff545d4297)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj" podUID="dfed6566-0e74-47e5-aa5f-edff545d4297"
Oct 30 22:30:09.177552 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:09.177532    4889 generic.go:356] "Generic (PLEG): container finished" podID="4091a3c0-1f2f-4444-ae6b-20fc34f19e4c" containerID="3cc54442170da199bb84e6939d5508068622770beb5ba871054f1d684aff4a17" exitCode=60
Oct 30 22:30:09.177660 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:09.177568    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd" event={"ID":"4091a3c0-1f2f-4444-ae6b-20fc34f19e4c","Type":"ContainerDied","Data":"3cc54442170da199bb84e6939d5508068622770beb5ba871054f1d684aff4a17"}
Oct 30 22:30:09.177947 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:09.177929    4889 scope.go:122] "RemoveContainer" containerID="3cc54442170da199bb84e6939d5508068622770beb5ba871054f1d684aff4a17"
Oct 30 22:30:09.178327 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:30:09.178303    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd_odh-ai-gateway-infra(4091a3c0-1f2f-4444-ae6b-20fc34f19e4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd" podUID="4091a3c0-1f2f-4444-ae6b-20fc34f19e4c"
Oct 30 22:30:09.180083 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:09.180064    4889 generic.go:356] "Generic (PLEG): container finished" podID="c9674a90-e299-4582-9375-0e6a9bed0e4a" containerID="eedac7c8f0c375595af10c59c17407f70acd97675f73edc2f6133a4f3e04a939" exitCode=60
Oct 30 22:30:09.180198 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:09.180132    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn" event={"ID":"c9674a90-e299-4582-9375-0e6a9bed0e4a","Type":"ContainerDied","Data":"eedac7c8f0c375595af10c59c17407f70acd97675f73edc2f6133a4f3e04a939"}
Oct 30 22:30:09.180400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:09.180382    4889 scope.go:122] "RemoveContainer" containerID="eedac7c8f0c375595af10c59c17407f70acd97675f73edc2f6133a4f3e04a939"
Oct 30 22:30:09.180669 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:30:09.180648    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn_odh-ai-gateway-infra(c9674a90-e299-4582-9375-0e6a9bed0e4a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn" podUID="c9674a90-e299-4582-9375-0e6a9bed0e4a"
Oct 30 22:30:09.190116 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:09.190095    4889 scope.go:122] "RemoveContainer" containerID="d0d1b81ce2bb49d412b56fded6f54a52d7a7927a75a4ad28bbeee10fad712d6d"
Oct 30 22:30:09.204945 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:09.204925    4889 scope.go:122] "RemoveContainer" containerID="3acceff5f01c86546721ef910654fa3ecb67f73086e54014f348dfaa71b3b141"
Oct 30 22:30:09.628217 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:09.628183    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:09.792460 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:09.792426    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"]
Oct 30 22:30:09.792720 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:09.792692    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr" podUID="b6ca58fb-2ebf-452a-a6d5-26311c091b1c" containerName="istio-proxy" containerID="cri-o://7cd10e61e23312ecc757569cf81ae5eece647767404f2ec36ebccf667df99436" gracePeriod=30
Oct 30 22:30:10.182246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:10.178419    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft"
Oct 30 22:30:10.290253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:10.270133    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:10.290253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:10.272588    4889 scope.go:122] "RemoveContainer" containerID="38a4d184c6d74bc75a7d33ce4b1525e82351c6520ceed4126bc5dd9ecf934768"
Oct 30 22:30:10.290253 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:30:10.273035    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-m6ptj_odh-ai-gateway-infra(dfed6566-0e74-47e5-aa5f-edff545d4297)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj" podUID="dfed6566-0e74-47e5-aa5f-edff545d4297"
Oct 30 22:30:10.431246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:10.430009    4889 scope.go:122] "RemoveContainer" containerID="3cc54442170da199bb84e6939d5508068622770beb5ba871054f1d684aff4a17"
Oct 30 22:30:10.431246 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:30:10.430513    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd_odh-ai-gateway-infra(4091a3c0-1f2f-4444-ae6b-20fc34f19e4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd" podUID="4091a3c0-1f2f-4444-ae6b-20fc34f19e4c"
Oct 30 22:30:10.469920 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:10.469837    4889 scope.go:122] "RemoveContainer" containerID="eedac7c8f0c375595af10c59c17407f70acd97675f73edc2f6133a4f3e04a939"
Oct 30 22:30:10.470577 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:30:10.470532    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn_odh-ai-gateway-infra(c9674a90-e299-4582-9375-0e6a9bed0e4a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn" podUID="c9674a90-e299-4582-9375-0e6a9bed0e4a"
Oct 30 22:30:10.475733 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:30:10.475699    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:30:10.564250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:10.557074    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:10.946248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:10.939933    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:11.120706 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:11.120676    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:12.904258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:12.899892    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:14.186444 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:14.186418    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9"
Oct 30 22:30:14.621121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:14.621084    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:15.109861 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.109830    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:30:15.225764 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.225680    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-podinfo\") pod \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") "
Oct 30 22:30:15.226263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.225780    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-envoy\") pod \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") "
Oct 30 22:30:15.226263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.225869    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-credential-socket\") pod \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") "
Oct 30 22:30:15.226263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.225910    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istiod-ca-cert\") pod \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") "
Oct 30 22:30:15.226263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.225972    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-token\" (UniqueName: \"kubernetes.io/projected/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-token\") pod \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") "
Oct 30 22:30:15.226263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.226169    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-data\") pod \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") "
Oct 30 22:30:15.226263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.226210    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-workload-socket\") pod \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") "
Oct 30 22:30:15.226621 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.226323    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-workload-certs\") pod \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") "
Oct 30 22:30:15.226621 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.226162    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-credential-socket" pod "b6ca58fb-2ebf-452a-a6d5-26311c091b1c" (UID: "b6ca58fb-2ebf-452a-a6d5-26311c091b1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:15.226621 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.226437    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-kube-api-access-vcskg\" (UniqueName: \"kubernetes.io/projected/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-kube-api-access-vcskg\") pod \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\" (UID: \"b6ca58fb-2ebf-452a-a6d5-26311c091b1c\") "
Oct 30 22:30:15.226621 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.226437    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-data" pod "b6ca58fb-2ebf-452a-a6d5-26311c091b1c" (UID: "b6ca58fb-2ebf-452a-a6d5-26311c091b1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:15.226621 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.226362    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istiod-ca-cert" pod "b6ca58fb-2ebf-452a-a6d5-26311c091b1c" (UID: "b6ca58fb-2ebf-452a-a6d5-26311c091b1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:15.226621 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.226552    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-workload-socket" pod "b6ca58fb-2ebf-452a-a6d5-26311c091b1c" (UID: "b6ca58fb-2ebf-452a-a6d5-26311c091b1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:15.226899 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.226666    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-workload-certs" pod "b6ca58fb-2ebf-452a-a6d5-26311c091b1c" (UID: "b6ca58fb-2ebf-452a-a6d5-26311c091b1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:15.227480 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.227450    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:30:15.227593 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.227484    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:30:15.227593 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.227501    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:30:15.227593 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.227514    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:30:15.227593 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.227528    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:30:15.228649 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.228621    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-envoy" pod "b6ca58fb-2ebf-452a-a6d5-26311c091b1c" (UID: "b6ca58fb-2ebf-452a-a6d5-26311c091b1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:15.228649 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.228629    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-podinfo" pod "b6ca58fb-2ebf-452a-a6d5-26311c091b1c" (UID: "b6ca58fb-2ebf-452a-a6d5-26311c091b1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:15.228854 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.228834    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-token" pod "b6ca58fb-2ebf-452a-a6d5-26311c091b1c" (UID: "b6ca58fb-2ebf-452a-a6d5-26311c091b1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:15.228905 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.228883    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-kube-api-access-vcskg" pod "b6ca58fb-2ebf-452a-a6d5-26311c091b1c" (UID: "b6ca58fb-2ebf-452a-a6d5-26311c091b1c"). InnerVolumeSpecName "kube-api-access-vcskg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:15.328763 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.328734    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:30:15.328763 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.328761    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcskg\" (UniqueName: \"kubernetes.io/projected/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-kube-api-access-vcskg\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:30:15.328949 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.328772    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:30:15.328949 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.328781    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6ca58fb-2ebf-452a-a6d5-26311c091b1c-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:30:15.508881 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.508853    4889 generic.go:356] "Generic (PLEG): container finished" podID="b6ca58fb-2ebf-452a-a6d5-26311c091b1c" containerID="7cd10e61e23312ecc757569cf81ae5eece647767404f2ec36ebccf667df99436" exitCode=0
Oct 30 22:30:15.509034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.508898    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"
Oct 30 22:30:15.509034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.508939    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr" event={"ID":"b6ca58fb-2ebf-452a-a6d5-26311c091b1c","Type":"ContainerDied","Data":"7cd10e61e23312ecc757569cf81ae5eece647767404f2ec36ebccf667df99436"}
Oct 30 22:30:15.509034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.509022    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr" event={"ID":"b6ca58fb-2ebf-452a-a6d5-26311c091b1c","Type":"ContainerDied","Data":"ccabc8b6ef653852d43ce5a251f6d6723d6348805ee0cc7182b290358a3585be"}
Oct 30 22:30:15.509151 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.509048    4889 scope.go:122] "RemoveContainer" containerID="7cd10e61e23312ecc757569cf81ae5eece647767404f2ec36ebccf667df99436"
Oct 30 22:30:15.522289 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.522266    4889 scope.go:122] "RemoveContainer" containerID="7cd10e61e23312ecc757569cf81ae5eece647767404f2ec36ebccf667df99436"
Oct 30 22:30:15.522584 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:30:15.522565    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cd10e61e23312ecc757569cf81ae5eece647767404f2ec36ebccf667df99436\": container with ID starting with 7cd10e61e23312ecc757569cf81ae5eece647767404f2ec36ebccf667df99436 not found: ID does not exist" containerID="7cd10e61e23312ecc757569cf81ae5eece647767404f2ec36ebccf667df99436"
Oct 30 22:30:15.522626 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.522593    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cd10e61e23312ecc757569cf81ae5eece647767404f2ec36ebccf667df99436"} err="failed to get container status \"7cd10e61e23312ecc757569cf81ae5eece647767404f2ec36ebccf667df99436\": rpc error: code = NotFound desc = could not find container \"7cd10e61e23312ecc757569cf81ae5eece647767404f2ec36ebccf667df99436\": container with ID starting with 7cd10e61e23312ecc757569cf81ae5eece647767404f2ec36ebccf667df99436 not found: ID does not exist"
Oct 30 22:30:15.560895 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.560862    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"]
Oct 30 22:30:15.571123 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:15.571097    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-xkmrr"]
Oct 30 22:30:16.434528 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:16.434043    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:16.778206 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:16.778170    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6ca58fb-2ebf-452a-a6d5-26311c091b1c" path="/var/lib/kubelet/pods/b6ca58fb-2ebf-452a-a6d5-26311c091b1c/volumes"
Oct 30 22:30:18.184255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:18.183386    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:18.399274 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:18.397697    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:18.577663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:18.577624    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:18.626959 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:18.626929    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"]
Oct 30 22:30:18.627144 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:18.627125    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9" podUID="eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820" containerName="istio-proxy" containerID="cri-o://2cbebb70dd8923a6a9ab9154a77f16b682f5f712df5a4141d784ba63a03be1e8" gracePeriod=30
Oct 30 22:30:18.766923 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:18.766891    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:19.522934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:19.522900    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:19.767175 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:19.767145    4889 scope.go:122] "RemoveContainer" containerID="3cc54442170da199bb84e6939d5508068622770beb5ba871054f1d684aff4a17"
Oct 30 22:30:19.965470 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:19.965440    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:20.185292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:20.185180    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:20.378066 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:20.378035    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:20.556397 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:20.556364    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd" event={"ID":"4091a3c0-1f2f-4444-ae6b-20fc34f19e4c","Type":"ContainerStarted","Data":"e19c9b0e292a846603ab58fffffcae8a544eef5bb5ddfc7a0b117844005dd54d"}
Oct 30 22:30:20.767365 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:20.767325    4889 scope.go:122] "RemoveContainer" containerID="38a4d184c6d74bc75a7d33ce4b1525e82351c6520ceed4126bc5dd9ecf934768"
Oct 30 22:30:21.566423 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:21.566387    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj" event={"ID":"dfed6566-0e74-47e5-aa5f-edff545d4297","Type":"ContainerStarted","Data":"f8e32d2bc608d0a98eb093450d6db6d5294c21a2516b3cba345b8a525f1d1576"}
Oct 30 22:30:21.568433 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:21.568403    4889 generic.go:356] "Generic (PLEG): container finished" podID="4091a3c0-1f2f-4444-ae6b-20fc34f19e4c" containerID="e19c9b0e292a846603ab58fffffcae8a544eef5bb5ddfc7a0b117844005dd54d" exitCode=60
Oct 30 22:30:21.568576 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:21.568475    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd" event={"ID":"4091a3c0-1f2f-4444-ae6b-20fc34f19e4c","Type":"ContainerDied","Data":"e19c9b0e292a846603ab58fffffcae8a544eef5bb5ddfc7a0b117844005dd54d"}
Oct 30 22:30:21.568576 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:21.568545    4889 scope.go:122] "RemoveContainer" containerID="3cc54442170da199bb84e6939d5508068622770beb5ba871054f1d684aff4a17"
Oct 30 22:30:21.568955 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:21.568938    4889 scope.go:122] "RemoveContainer" containerID="e19c9b0e292a846603ab58fffffcae8a544eef5bb5ddfc7a0b117844005dd54d"
Oct 30 22:30:21.569371 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:30:21.569344    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd_odh-ai-gateway-infra(4091a3c0-1f2f-4444-ae6b-20fc34f19e4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd" podUID="4091a3c0-1f2f-4444-ae6b-20fc34f19e4c"
Oct 30 22:30:21.612801 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:21.612766    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd"]
Oct 30 22:30:21.958878 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:21.951174    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:22.579261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:22.579209    4889 generic.go:356] "Generic (PLEG): container finished" podID="dfed6566-0e74-47e5-aa5f-edff545d4297" containerID="f8e32d2bc608d0a98eb093450d6db6d5294c21a2516b3cba345b8a525f1d1576" exitCode=60
Oct 30 22:30:22.579737 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:22.579257    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj" event={"ID":"dfed6566-0e74-47e5-aa5f-edff545d4297","Type":"ContainerDied","Data":"f8e32d2bc608d0a98eb093450d6db6d5294c21a2516b3cba345b8a525f1d1576"}
Oct 30 22:30:22.579737 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:22.579363    4889 scope.go:122] "RemoveContainer" containerID="38a4d184c6d74bc75a7d33ce4b1525e82351c6520ceed4126bc5dd9ecf934768"
Oct 30 22:30:22.579845 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:22.579820    4889 scope.go:122] "RemoveContainer" containerID="f8e32d2bc608d0a98eb093450d6db6d5294c21a2516b3cba345b8a525f1d1576"
Oct 30 22:30:22.580292 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:30:22.580208    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-m6ptj_odh-ai-gateway-infra(dfed6566-0e74-47e5-aa5f-edff545d4297)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj" podUID="dfed6566-0e74-47e5-aa5f-edff545d4297"
Oct 30 22:30:22.633538 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:22.633509    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj"]
Oct 30 22:30:22.767585 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:22.767556    4889 scope.go:122] "RemoveContainer" containerID="eedac7c8f0c375595af10c59c17407f70acd97675f73edc2f6133a4f3e04a939"
Oct 30 22:30:22.832040 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:22.831980    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd"
Oct 30 22:30:22.916744 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:22.916697    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4091a3c0-1f2f-4444-ae6b-20fc34f19e4c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4091a3c0-1f2f-4444-ae6b-20fc34f19e4c-maas-api-service-ca\") pod \"4091a3c0-1f2f-4444-ae6b-20fc34f19e4c\" (UID: \"4091a3c0-1f2f-4444-ae6b-20fc34f19e4c\") "
Oct 30 22:30:22.916940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:22.916889    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4091a3c0-1f2f-4444-ae6b-20fc34f19e4c-kube-api-access-s4422\" (UniqueName: \"kubernetes.io/projected/4091a3c0-1f2f-4444-ae6b-20fc34f19e4c-kube-api-access-s4422\") pod \"4091a3c0-1f2f-4444-ae6b-20fc34f19e4c\" (UID: \"4091a3c0-1f2f-4444-ae6b-20fc34f19e4c\") "
Oct 30 22:30:22.917154 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:22.917132    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4091a3c0-1f2f-4444-ae6b-20fc34f19e4c-maas-api-service-ca" pod "4091a3c0-1f2f-4444-ae6b-20fc34f19e4c" (UID: "4091a3c0-1f2f-4444-ae6b-20fc34f19e4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:22.917544 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:22.917521    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4091a3c0-1f2f-4444-ae6b-20fc34f19e4c-maas-api-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:30:23.009994 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:23.009958    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4091a3c0-1f2f-4444-ae6b-20fc34f19e4c-kube-api-access-s4422" pod "4091a3c0-1f2f-4444-ae6b-20fc34f19e4c" (UID: "4091a3c0-1f2f-4444-ae6b-20fc34f19e4c"). InnerVolumeSpecName "kube-api-access-s4422". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:23.018972 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:23.018945    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4422\" (UniqueName: \"kubernetes.io/projected/4091a3c0-1f2f-4444-ae6b-20fc34f19e4c-kube-api-access-s4422\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:30:23.839261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:23.833942    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd" event={"ID":"4091a3c0-1f2f-4444-ae6b-20fc34f19e4c","Type":"ContainerDied","Data":"38b983d331ff4cab2b10fd1beb9f8c0170f2d3d583fecdb9f70bb6df7adc3347"}
Oct 30 22:30:23.839261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:23.834030    4889 scope.go:122] "RemoveContainer" containerID="e19c9b0e292a846603ab58fffffcae8a544eef5bb5ddfc7a0b117844005dd54d"
Oct 30 22:30:23.839261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:23.834137    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd"
Oct 30 22:30:23.903292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:23.894520    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn" event={"ID":"c9674a90-e299-4582-9375-0e6a9bed0e4a","Type":"ContainerStarted","Data":"ca940321cc2dafe4e42e8d96f51f5027d82c615915ae0e1abc76c364e81190a8"}
Oct 30 22:30:23.903292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:23.895259    4889 scope.go:122] "RemoveContainer" containerID="ca940321cc2dafe4e42e8d96f51f5027d82c615915ae0e1abc76c364e81190a8"
Oct 30 22:30:23.903292 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:30:23.895756    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn_odh-ai-gateway-infra(c9674a90-e299-4582-9375-0e6a9bed0e4a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn" podUID="c9674a90-e299-4582-9375-0e6a9bed0e4a"
Oct 30 22:30:23.963274 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:23.961316    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd"]
Oct 30 22:30:23.964395 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:23.963885    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-370ec2-29889990-cpftd"]
Oct 30 22:30:24.070373 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.070202    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj"
Oct 30 22:30:24.137506 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.137480    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:30:24.202748 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.202658    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:24.212322 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.212293    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfed6566-0e74-47e5-aa5f-edff545d4297-kube-api-access-6ff76\" (UniqueName: \"kubernetes.io/projected/dfed6566-0e74-47e5-aa5f-edff545d4297-kube-api-access-6ff76\") pod \"dfed6566-0e74-47e5-aa5f-edff545d4297\" (UID: \"dfed6566-0e74-47e5-aa5f-edff545d4297\") "
Oct 30 22:30:24.227293 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.219370    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfed6566-0e74-47e5-aa5f-edff545d4297-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfed6566-0e74-47e5-aa5f-edff545d4297-maas-api-service-ca\") pod \"dfed6566-0e74-47e5-aa5f-edff545d4297\" (UID: \"dfed6566-0e74-47e5-aa5f-edff545d4297\") "
Oct 30 22:30:24.227293 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.219893    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfed6566-0e74-47e5-aa5f-edff545d4297-maas-api-service-ca" pod "dfed6566-0e74-47e5-aa5f-edff545d4297" (UID: "dfed6566-0e74-47e5-aa5f-edff545d4297"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:24.227293 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.220612    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfed6566-0e74-47e5-aa5f-edff545d4297-maas-api-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.227293 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.226464    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfed6566-0e74-47e5-aa5f-edff545d4297-kube-api-access-6ff76" pod "dfed6566-0e74-47e5-aa5f-edff545d4297" (UID: "dfed6566-0e74-47e5-aa5f-edff545d4297"). InnerVolumeSpecName "kube-api-access-6ff76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:24.332262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.330342    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-podinfo\") pod \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") "
Oct 30 22:30:24.332262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.330586    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-credential-socket\") pod \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") "
Oct 30 22:30:24.332262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.330628    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-envoy\") pod \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") "
Oct 30 22:30:24.332262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.330693    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-kube-api-access-l9p94\" (UniqueName: \"kubernetes.io/projected/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-kube-api-access-l9p94\") pod \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") "
Oct 30 22:30:24.332262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.330751    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istiod-ca-cert\") pod \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") "
Oct 30 22:30:24.332262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.330856    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-workload-socket\") pod \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") "
Oct 30 22:30:24.332262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.331027    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-token\" (UniqueName: \"kubernetes.io/projected/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-token\") pod \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") "
Oct 30 22:30:24.332262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.331070    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-workload-certs\") pod \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") "
Oct 30 22:30:24.332262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.331155    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-data\") pod \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\" (UID: \"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820\") "
Oct 30 22:30:24.332262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.332104    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ff76\" (UniqueName: \"kubernetes.io/projected/dfed6566-0e74-47e5-aa5f-edff545d4297-kube-api-access-6ff76\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.339091 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.339058    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-workload-socket" pod "eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820" (UID: "eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:24.339622 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.339599    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istiod-ca-cert" pod "eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820" (UID: "eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:24.365249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.355323    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-data" pod "eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820" (UID: "eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:24.365249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.363714    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-envoy" pod "eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820" (UID: "eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:24.365612 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.365583    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-workload-certs" pod "eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820" (UID: "eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:24.365769 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.365751    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-credential-socket" pod "eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820" (UID: "eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:24.367803 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.367774    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-token" pod "eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820" (UID: "eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:24.380751 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.380687    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-podinfo" pod "eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820" (UID: "eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:24.381707 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.381661    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-kube-api-access-l9p94" pod "eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820" (UID: "eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820"). InnerVolumeSpecName "kube-api-access-l9p94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:24.417081 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.416883    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:24.436479 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.435937    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.436479 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.435975    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.436479 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.435988    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.436479 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.436002    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.436479 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.436015    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.436479 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.436027    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.436479 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.436039    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.436479 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.436053    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.436479 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.436068    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9p94\" (UniqueName: \"kubernetes.io/projected/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820-kube-api-access-l9p94\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.781487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.781451    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4091a3c0-1f2f-4444-ae6b-20fc34f19e4c" path="/var/lib/kubelet/pods/4091a3c0-1f2f-4444-ae6b-20fc34f19e4c/volumes"
Oct 30 22:30:24.904028 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.904002    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj"
Oct 30 22:30:24.904028 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.904013    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj" event={"ID":"dfed6566-0e74-47e5-aa5f-edff545d4297","Type":"ContainerDied","Data":"6f2a48dae6f1999babf82bb7397e82fcb87deae34b7c55cdf86298f57d204f5c"}
Oct 30 22:30:24.904559 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.904075    4889 scope.go:122] "RemoveContainer" containerID="f8e32d2bc608d0a98eb093450d6db6d5294c21a2516b3cba345b8a525f1d1576"
Oct 30 22:30:24.910078 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.910054    4889 generic.go:356] "Generic (PLEG): container finished" podID="eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820" containerID="2cbebb70dd8923a6a9ab9154a77f16b682f5f712df5a4141d784ba63a03be1e8" exitCode=0
Oct 30 22:30:24.910202 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.910095    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"
Oct 30 22:30:24.910202 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.910149    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9" event={"ID":"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820","Type":"ContainerDied","Data":"2cbebb70dd8923a6a9ab9154a77f16b682f5f712df5a4141d784ba63a03be1e8"}
Oct 30 22:30:24.910202 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.910192    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9" event={"ID":"eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820","Type":"ContainerDied","Data":"ec58b242c96ee02e9f79b77a41fd3dc1e1b1a5aa239cb26cb3b3badef2c5ebbc"}
Oct 30 22:30:24.912363 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.912338    4889 generic.go:356] "Generic (PLEG): container finished" podID="c9674a90-e299-4582-9375-0e6a9bed0e4a" containerID="ca940321cc2dafe4e42e8d96f51f5027d82c615915ae0e1abc76c364e81190a8" exitCode=60
Oct 30 22:30:24.912472 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.912405    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn" event={"ID":"c9674a90-e299-4582-9375-0e6a9bed0e4a","Type":"ContainerDied","Data":"ca940321cc2dafe4e42e8d96f51f5027d82c615915ae0e1abc76c364e81190a8"}
Oct 30 22:30:24.912826 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.912809    4889 scope.go:122] "RemoveContainer" containerID="ca940321cc2dafe4e42e8d96f51f5027d82c615915ae0e1abc76c364e81190a8"
Oct 30 22:30:24.913109 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:30:24.913085    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn_odh-ai-gateway-infra(c9674a90-e299-4582-9375-0e6a9bed0e4a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn" podUID="c9674a90-e299-4582-9375-0e6a9bed0e4a"
Oct 30 22:30:24.916954 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.916933    4889 scope.go:122] "RemoveContainer" containerID="2cbebb70dd8923a6a9ab9154a77f16b682f5f712df5a4141d784ba63a03be1e8"
Oct 30 22:30:24.935358 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.935333    4889 scope.go:122] "RemoveContainer" containerID="2cbebb70dd8923a6a9ab9154a77f16b682f5f712df5a4141d784ba63a03be1e8"
Oct 30 22:30:24.935699 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:30:24.935673    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cbebb70dd8923a6a9ab9154a77f16b682f5f712df5a4141d784ba63a03be1e8\": container with ID starting with 2cbebb70dd8923a6a9ab9154a77f16b682f5f712df5a4141d784ba63a03be1e8 not found: ID does not exist" containerID="2cbebb70dd8923a6a9ab9154a77f16b682f5f712df5a4141d784ba63a03be1e8"
Oct 30 22:30:24.935772 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.935712    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cbebb70dd8923a6a9ab9154a77f16b682f5f712df5a4141d784ba63a03be1e8"} err="failed to get container status \"2cbebb70dd8923a6a9ab9154a77f16b682f5f712df5a4141d784ba63a03be1e8\": rpc error: code = NotFound desc = could not find container \"2cbebb70dd8923a6a9ab9154a77f16b682f5f712df5a4141d784ba63a03be1e8\": container with ID starting with 2cbebb70dd8923a6a9ab9154a77f16b682f5f712df5a4141d784ba63a03be1e8 not found: ID does not exist"
Oct 30 22:30:24.935772 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.935736    4889 scope.go:122] "RemoveContainer" containerID="eedac7c8f0c375595af10c59c17407f70acd97675f73edc2f6133a4f3e04a939"
Oct 30 22:30:24.972316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.972290    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj"]
Oct 30 22:30:24.978869 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:24.978846    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6ptj"]
Oct 30 22:30:25.003623 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:25.003599    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn"]
Oct 30 22:30:25.097272 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:25.097162    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"]
Oct 30 22:30:25.102668 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:25.102643    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-k9sr9"]
Oct 30 22:30:25.516125 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:25.516075    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:25.950123 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:25.950089    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:26.176301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:26.175263    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:26.291244 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:26.276134    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn"
Oct 30 22:30:26.312338 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:26.308313    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9674a90-e299-4582-9375-0e6a9bed0e4a-kube-api-access-7zb6m\" (UniqueName: \"kubernetes.io/projected/c9674a90-e299-4582-9375-0e6a9bed0e4a-kube-api-access-7zb6m\") pod \"c9674a90-e299-4582-9375-0e6a9bed0e4a\" (UID: \"c9674a90-e299-4582-9375-0e6a9bed0e4a\") "
Oct 30 22:30:26.312338 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:26.308379    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9674a90-e299-4582-9375-0e6a9bed0e4a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9674a90-e299-4582-9375-0e6a9bed0e4a-maas-api-service-ca\") pod \"c9674a90-e299-4582-9375-0e6a9bed0e4a\" (UID: \"c9674a90-e299-4582-9375-0e6a9bed0e4a\") "
Oct 30 22:30:26.312338 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:26.309959    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9674a90-e299-4582-9375-0e6a9bed0e4a-maas-api-service-ca" pod "c9674a90-e299-4582-9375-0e6a9bed0e4a" (UID: "c9674a90-e299-4582-9375-0e6a9bed0e4a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:26.327277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:26.315758    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9674a90-e299-4582-9375-0e6a9bed0e4a-kube-api-access-7zb6m" pod "c9674a90-e299-4582-9375-0e6a9bed0e4a" (UID: "c9674a90-e299-4582-9375-0e6a9bed0e4a"). InnerVolumeSpecName "kube-api-access-7zb6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:26.418753 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:26.417149    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zb6m\" (UniqueName: \"kubernetes.io/projected/c9674a90-e299-4582-9375-0e6a9bed0e4a-kube-api-access-7zb6m\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:30:26.418753 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:26.417187    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9674a90-e299-4582-9375-0e6a9bed0e4a-maas-api-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:30:26.429291 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:26.419970    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:26.750715 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:26.750678    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:26.839356 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:26.839320    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfed6566-0e74-47e5-aa5f-edff545d4297" path="/var/lib/kubelet/pods/dfed6566-0e74-47e5-aa5f-edff545d4297/volumes"
Oct 30 22:30:26.887717 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:26.887683    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820" path="/var/lib/kubelet/pods/eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820/volumes"
Oct 30 22:30:26.977247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:26.973961    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:27.015907 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:27.009297    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn" event={"ID":"c9674a90-e299-4582-9375-0e6a9bed0e4a","Type":"ContainerDied","Data":"b60ec17ff9a9982823459e2a1014949e63cf45cf01f7e05f7fa26ebcbc6947c6"}
Oct 30 22:30:27.015907 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:27.009371    4889 scope.go:122] "RemoveContainer" containerID="ca940321cc2dafe4e42e8d96f51f5027d82c615915ae0e1abc76c364e81190a8"
Oct 30 22:30:27.015907 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:27.009488    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn"
Oct 30 22:30:27.090251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:27.089525    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn"]
Oct 30 22:30:27.162279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:27.161394    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29889990-fxwcn"]
Oct 30 22:30:28.828247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:28.823506    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9674a90-e299-4582-9375-0e6a9bed0e4a" path="/var/lib/kubelet/pods/c9674a90-e299-4582-9375-0e6a9bed0e4a/volumes"
Oct 30 22:30:29.657374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:29.656852    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:29.882107 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:29.879084    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:30.106251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:30.097775    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:31.457103 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:31.457055    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:31.669104 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:31.669054    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:32.352148 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:32.352100    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:33.266609 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.266584    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:33.370492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.370459    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6"]
Oct 30 22:30:33.371982 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.371945    4889 cpu_manager.go:414] "containerMap: removing container" podUID="eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820" containerName="istio-proxy"
Oct 30 22:30:33.371982 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.371977    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820" containerName="istio-proxy"
Oct 30 22:30:33.372193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.371998    4889 cpu_manager.go:414] "containerMap: removing container" podUID="dfed6566-0e74-47e5-aa5f-edff545d4297" containerName="cleanup"
Oct 30 22:30:33.372193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.372007    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfed6566-0e74-47e5-aa5f-edff545d4297" containerName="cleanup"
Oct 30 22:30:33.372193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.372025    4889 cpu_manager.go:414] "containerMap: removing container" podUID="4091a3c0-1f2f-4444-ae6b-20fc34f19e4c" containerName="cleanup"
Oct 30 22:30:33.372193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.372032    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4091a3c0-1f2f-4444-ae6b-20fc34f19e4c" containerName="cleanup"
Oct 30 22:30:33.372193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.372041    4889 cpu_manager.go:414] "containerMap: removing container" podUID="dfed6566-0e74-47e5-aa5f-edff545d4297" containerName="cleanup"
Oct 30 22:30:33.372193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.372048    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfed6566-0e74-47e5-aa5f-edff545d4297" containerName="cleanup"
Oct 30 22:30:33.372193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.372063    4889 cpu_manager.go:414] "containerMap: removing container" podUID="c9674a90-e299-4582-9375-0e6a9bed0e4a" containerName="cleanup"
Oct 30 22:30:33.372193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.372070    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9674a90-e299-4582-9375-0e6a9bed0e4a" containerName="cleanup"
Oct 30 22:30:33.372193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.372096    4889 cpu_manager.go:414] "containerMap: removing container" podUID="b6ca58fb-2ebf-452a-a6d5-26311c091b1c" containerName="istio-proxy"
Oct 30 22:30:33.372193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.372103    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6ca58fb-2ebf-452a-a6d5-26311c091b1c" containerName="istio-proxy"
Oct 30 22:30:33.372193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.372116    4889 cpu_manager.go:414] "containerMap: removing container" podUID="4091a3c0-1f2f-4444-ae6b-20fc34f19e4c" containerName="cleanup"
Oct 30 22:30:33.372193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.372123    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4091a3c0-1f2f-4444-ae6b-20fc34f19e4c" containerName="cleanup"
Oct 30 22:30:33.372193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.372133    4889 cpu_manager.go:414] "containerMap: removing container" podUID="c9674a90-e299-4582-9375-0e6a9bed0e4a" containerName="cleanup"
Oct 30 22:30:33.372193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.372140    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9674a90-e299-4582-9375-0e6a9bed0e4a" containerName="cleanup"
Oct 30 22:30:33.372811 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.372366    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="4091a3c0-1f2f-4444-ae6b-20fc34f19e4c" containerName="cleanup"
Oct 30 22:30:33.372811 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.372381    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfed6566-0e74-47e5-aa5f-edff545d4297" containerName="cleanup"
Oct 30 22:30:33.372811 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.372391    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9674a90-e299-4582-9375-0e6a9bed0e4a" containerName="cleanup"
Oct 30 22:30:33.372811 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.372403    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6ca58fb-2ebf-452a-a6d5-26311c091b1c" containerName="istio-proxy"
Oct 30 22:30:33.372811 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.372414    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="4091a3c0-1f2f-4444-ae6b-20fc34f19e4c" containerName="cleanup"
Oct 30 22:30:33.372811 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.372428    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="4091a3c0-1f2f-4444-ae6b-20fc34f19e4c" containerName="cleanup"
Oct 30 22:30:33.372811 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.372439    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9674a90-e299-4582-9375-0e6a9bed0e4a" containerName="cleanup"
Oct 30 22:30:33.372811 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.372451    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb0ab0bb-35e8-4e4e-8f79-35cfd6e56820" containerName="istio-proxy"
Oct 30 22:30:33.372811 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.372461    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfed6566-0e74-47e5-aa5f-edff545d4297" containerName="cleanup"
Oct 30 22:30:33.382070 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.382041    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6"]
Oct 30 22:30:33.382320 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.382299    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6"
Oct 30 22:30:33.447268 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.433059    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-bf9c2f\""
Oct 30 22:30:33.520320 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.515521    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-bf9c2f-dockercfg-qjdpl\""
Oct 30 22:30:33.520320 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.517688    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfkts\" (UniqueName: \"kubernetes.io/projected/40454feb-b149-4312-a275-62e8dbac9c23-kube-api-access-cfkts\") pod \"payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6\" (UID: \"40454feb-b149-4312-a275-62e8dbac9c23\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6"
Oct 30 22:30:33.520320 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.517975    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40454feb-b149-4312-a275-62e8dbac9c23-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6\" (UID: \"40454feb-b149-4312-a275-62e8dbac9c23\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6"
Oct 30 22:30:33.544597 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.544535    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:33.571385 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.571289    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq"]
Oct 30 22:30:33.572939 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.572909    4889 cpu_manager.go:414] "containerMap: removing container" podUID="dfed6566-0e74-47e5-aa5f-edff545d4297" containerName="cleanup"
Oct 30 22:30:33.573092 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.573080    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfed6566-0e74-47e5-aa5f-edff545d4297" containerName="cleanup"
Oct 30 22:30:33.573208 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.573198    4889 cpu_manager.go:414] "containerMap: removing container" podUID="c9674a90-e299-4582-9375-0e6a9bed0e4a" containerName="cleanup"
Oct 30 22:30:33.577302 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.577274    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9674a90-e299-4582-9375-0e6a9bed0e4a" containerName="cleanup"
Oct 30 22:30:33.577545 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.577530    4889 cpu_manager.go:414] "containerMap: removing container" podUID="4091a3c0-1f2f-4444-ae6b-20fc34f19e4c" containerName="cleanup"
Oct 30 22:30:33.577646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.577635    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4091a3c0-1f2f-4444-ae6b-20fc34f19e4c" containerName="cleanup"
Oct 30 22:30:33.584265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.577956    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfed6566-0e74-47e5-aa5f-edff545d4297" containerName="cleanup"
Oct 30 22:30:33.584546 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.584520    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9674a90-e299-4582-9375-0e6a9bed0e4a" containerName="cleanup"
Oct 30 22:30:33.609407 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.609364    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq"
Oct 30 22:30:33.669270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.621364    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfkts\" (UniqueName: \"kubernetes.io/projected/40454feb-b149-4312-a275-62e8dbac9c23-kube-api-access-cfkts\") pod \"payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6\" (UID: \"40454feb-b149-4312-a275-62e8dbac9c23\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6"
Oct 30 22:30:33.669270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.621687    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40454feb-b149-4312-a275-62e8dbac9c23-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6\" (UID: \"40454feb-b149-4312-a275-62e8dbac9c23\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6"
Oct 30 22:30:33.669270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.622635    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40454feb-b149-4312-a275-62e8dbac9c23-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6\" (UID: \"40454feb-b149-4312-a275-62e8dbac9c23\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6"
Oct 30 22:30:33.735249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.730919    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d3de30f-5af9-4195-bb1b-1de3f0643e06-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq\" (UID: \"6d3de30f-5af9-4195-bb1b-1de3f0643e06\") " pod="openshift-ingress/payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq"
Oct 30 22:30:33.792313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.739490    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v74ld\" (UniqueName: \"kubernetes.io/projected/6d3de30f-5af9-4195-bb1b-1de3f0643e06-kube-api-access-v74ld\") pod \"payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq\" (UID: \"6d3de30f-5af9-4195-bb1b-1de3f0643e06\") " pod="openshift-ingress/payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq"
Oct 30 22:30:33.874150 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.793087    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfkts\" (UniqueName: \"kubernetes.io/projected/40454feb-b149-4312-a275-62e8dbac9c23-kube-api-access-cfkts\") pod \"payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6\" (UID: \"40454feb-b149-4312-a275-62e8dbac9c23\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6"
Oct 30 22:30:33.899330 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.899283    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq"]
Oct 30 22:30:33.975214 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.975186    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d3de30f-5af9-4195-bb1b-1de3f0643e06-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq\" (UID: \"6d3de30f-5af9-4195-bb1b-1de3f0643e06\") " pod="openshift-ingress/payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq"
Oct 30 22:30:33.975645 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.975628    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v74ld\" (UniqueName: \"kubernetes.io/projected/6d3de30f-5af9-4195-bb1b-1de3f0643e06-kube-api-access-v74ld\") pod \"payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq\" (UID: \"6d3de30f-5af9-4195-bb1b-1de3f0643e06\") " pod="openshift-ingress/payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq"
Oct 30 22:30:33.976177 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.976156    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d3de30f-5af9-4195-bb1b-1de3f0643e06-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq\" (UID: \"6d3de30f-5af9-4195-bb1b-1de3f0643e06\") " pod="openshift-ingress/payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq"
Oct 30 22:30:33.994618 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:33.994590    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v74ld\" (UniqueName: \"kubernetes.io/projected/6d3de30f-5af9-4195-bb1b-1de3f0643e06-kube-api-access-v74ld\") pod \"payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq\" (UID: \"6d3de30f-5af9-4195-bb1b-1de3f0643e06\") " pod="openshift-ingress/payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq"
Oct 30 22:30:34.028575 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:34.028528    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6"
Oct 30 22:30:34.185792 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:34.185556    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq"
Oct 30 22:30:34.471752 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:34.463930    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6"]
Oct 30 22:30:34.617302 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:34.609429    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq"]
Oct 30 22:30:34.639252 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:30:34.609218    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3de30f_5af9_4195_bb1b_1de3f0643e06.slice/crio-e3e09aa9e1ab5168a9416d506ac3abaaabea7ae1b1537e2b2cdaacb33ca10ad8 WatchSource:0}: Error finding container e3e09aa9e1ab5168a9416d506ac3abaaabea7ae1b1537e2b2cdaacb33ca10ad8: Status 404 returned error can't find the container with id e3e09aa9e1ab5168a9416d506ac3abaaabea7ae1b1537e2b2cdaacb33ca10ad8
Oct 30 22:30:34.972252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:34.971341    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:35.395795 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:35.395763    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq" event={"ID":"6d3de30f-5af9-4195-bb1b-1de3f0643e06","Type":"ContainerStarted","Data":"55f97ddae8ee5f2ee9ff2a61403960ecd5e006882fbe61d12f5f20834e417d29"}
Oct 30 22:30:35.395983 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:35.395835    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq" event={"ID":"6d3de30f-5af9-4195-bb1b-1de3f0643e06","Type":"ContainerStarted","Data":"e3e09aa9e1ab5168a9416d506ac3abaaabea7ae1b1537e2b2cdaacb33ca10ad8"}
Oct 30 22:30:35.396092 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:35.396031    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq"
Oct 30 22:30:35.398019 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:35.397506    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6" event={"ID":"40454feb-b149-4312-a275-62e8dbac9c23","Type":"ContainerStarted","Data":"1f6bd96dc1f4358e233171409c4501453ba23025397870c3bec30fae9b1571a1"}
Oct 30 22:30:35.398019 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:35.397567    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6"
Oct 30 22:30:35.398019 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:35.397584    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6" event={"ID":"40454feb-b149-4312-a275-62e8dbac9c23","Type":"ContainerStarted","Data":"2da3336ddc6c293ec4eab9f623a2c10afb008bdce78f4b190a97445e29a9dbf4"}
Oct 30 22:30:35.460252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:35.460033    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq" podStartSLOduration=2.4600179300000002 podStartE2EDuration="2.46001793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:35.439492669 +0000 UTC m=+1420.795858983" watchObservedRunningTime="2026-10-30 22:30:35.46001793 +0000 UTC m=+1420.816384265"
Oct 30 22:30:35.485291 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:35.485242    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6" podStartSLOduration=2.485214747 podStartE2EDuration="2.485214747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:35.48398279 +0000 UTC m=+1420.840349124" watchObservedRunningTime="2026-10-30 22:30:35.485214747 +0000 UTC m=+1420.841581061"
Oct 30 22:30:37.116245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:37.116159    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:38.968074 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:38.968042    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:40.309911 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:30:40.309867    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:30:40.445519 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.445441    4889 generic.go:356] "Generic (PLEG): container finished" podID="7cfa769f-0310-4749-a929-646c9ab9bea5" containerID="fcd12a6635b21bdca58d709888fee4cd88f8890fa114044e2004c2c4ef80c411" exitCode=0
Oct 30 22:30:40.445700 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.445514    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h" event={"ID":"7cfa769f-0310-4749-a929-646c9ab9bea5","Type":"ContainerDied","Data":"fcd12a6635b21bdca58d709888fee4cd88f8890fa114044e2004c2c4ef80c411"}
Oct 30 22:30:40.445986 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.445969    4889 scope.go:122] "RemoveContainer" containerID="fcd12a6635b21bdca58d709888fee4cd88f8890fa114044e2004c2c4ef80c411"
Oct 30 22:30:40.447788 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.447758    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:30:40.447886 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.447826    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:30:40.447886 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.447858    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:30:40.577506 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.577454    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"]
Oct 30 22:30:40.582143 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.582121    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.654284 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.653834    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"]
Oct 30 22:30:40.748996 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.748966    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29631b37-3676-4395-a55d-f09f9b3e0025-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.749157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.749017    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.749157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.749122    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.749157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.749153    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29631b37-3676-4395-a55d-f09f9b3e0025-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.749297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.749171    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29631b37-3676-4395-a55d-f09f9b3e0025-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.749382 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.749365    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.749419 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.749402    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.749454 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.749434    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhplt\" (UniqueName: \"kubernetes.io/projected/29631b37-3676-4395-a55d-f09f9b3e0025-kube-api-access-lhplt\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.749602 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.749579    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.851528 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.851496    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.851702 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.851542    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.851702 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.851585    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhplt\" (UniqueName: \"kubernetes.io/projected/29631b37-3676-4395-a55d-f09f9b3e0025-kube-api-access-lhplt\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.851702 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.851615    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.851702 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.851685    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29631b37-3676-4395-a55d-f09f9b3e0025-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.852135 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.851729    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.852135 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.851796    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.852135 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.851845    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29631b37-3676-4395-a55d-f09f9b3e0025-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.852135 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.851872    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29631b37-3676-4395-a55d-f09f9b3e0025-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.852135 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.851929    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.852135 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.852086    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.852598 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.852194    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.852598 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.852404    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.852705 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.852662    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29631b37-3676-4395-a55d-f09f9b3e0025-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.854312 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.854290    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.854803 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.854782    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29631b37-3676-4395-a55d-f09f9b3e0025-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.871005 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.870976    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29631b37-3676-4395-a55d-f09f9b3e0025-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.871256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.871214    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhplt\" (UniqueName: \"kubernetes.io/projected/29631b37-3676-4395-a55d-f09f9b3e0025-kube-api-access-lhplt\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:40.901604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:40.901576    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:41.066327 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:41.049490    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:41.223545 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:30:41.223515    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29631b37_3676_4395_a55d_f09f9b3e0025.slice/crio-5ce0e743d1f195655df4f44f95c29ebced991f3af4e110285a6531800456114f WatchSource:0}: Error finding container 5ce0e743d1f195655df4f44f95c29ebced991f3af4e110285a6531800456114f: Status 404 returned error can't find the container with id 5ce0e743d1f195655df4f44f95c29ebced991f3af4e110285a6531800456114f
Oct 30 22:30:41.226769 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:41.226727    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:30:41.226864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:41.226816    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:30:41.226864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:41.226858    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:30:41.227090 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:41.227072    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"]
Oct 30 22:30:41.229458 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:41.229419    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:41.458860 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:41.458783    4889 generic.go:356] "Generic (PLEG): container finished" podID="cf853efa-c753-4ce6-834d-c315fba26177" containerID="e6213e7a04157ee84a7d9dc781c8bc7e201caac96e85d397e91060fa41908c57" exitCode=0
Oct 30 22:30:41.459214 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:41.458863    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t" event={"ID":"cf853efa-c753-4ce6-834d-c315fba26177","Type":"ContainerDied","Data":"e6213e7a04157ee84a7d9dc781c8bc7e201caac96e85d397e91060fa41908c57"}
Oct 30 22:30:41.459369 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:41.459347    4889 scope.go:122] "RemoveContainer" containerID="e6213e7a04157ee84a7d9dc781c8bc7e201caac96e85d397e91060fa41908c57"
Oct 30 22:30:41.462200 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:41.462165    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:30:41.462319 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:41.462290    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:30:41.462370 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:41.462344    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:30:41.463360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:41.463336    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h" event={"ID":"7cfa769f-0310-4749-a929-646c9ab9bea5","Type":"ContainerStarted","Data":"64c2ef9d16cd8cf15f1e6a5ef4ac7ebfe4753234b9a225c85047034b91bf4682"}
Oct 30 22:30:41.465357 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:41.465336    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh" event={"ID":"29631b37-3676-4395-a55d-f09f9b3e0025","Type":"ContainerStarted","Data":"81a7d52cdd909cd00bcba9e276b378ab0baeb3a27f535e841886401174e2459b"}
Oct 30 22:30:41.465460 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:41.465374    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh" event={"ID":"29631b37-3676-4395-a55d-f09f9b3e0025","Type":"ContainerStarted","Data":"5ce0e743d1f195655df4f44f95c29ebced991f3af4e110285a6531800456114f"}
Oct 30 22:30:41.651298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:41.650246    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh" podStartSLOduration=1.650204409 podStartE2EDuration="1.650204409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:41.622714314 +0000 UTC m=+1426.979080628" watchObservedRunningTime="2026-10-30 22:30:41.650204409 +0000 UTC m=+1427.006570722"
Oct 30 22:30:41.747737 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:41.747700    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:41.920249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:41.917328    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:41.944258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:41.941871    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:42.114296 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:42.103777    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:30:42.114296 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:42.103893    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 22:30:42.114296 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:42.103935    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h" podUID="7cfa769f-0310-4749-a929-646c9ab9bea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 22:30:42.481861 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:42.481783    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t" event={"ID":"cf853efa-c753-4ce6-834d-c315fba26177","Type":"ContainerStarted","Data":"0e6e14c1e558127bda6e7dde7960e9a93c3dab845c6761f24e46a63b131307ab"}
Oct 30 22:30:42.908248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:42.905628    4889 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 22:30:42.908248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:42.905711    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh" podUID="29631b37-3676-4395-a55d-f09f9b3e0025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 22:30:42.956249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:42.955747    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:30:42.969252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:42.958322    4889 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 22:30:42.969252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:42.958377    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t" podUID="cf853efa-c753-4ce6-834d-c315fba26177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 22:30:43.103562 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:43.103526    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 22:30:43.103760 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:43.103599    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h" podUID="7cfa769f-0310-4749-a929-646c9ab9bea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 22:30:43.236304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:43.235989    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:43.575254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:43.566329    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:30:43.905032 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:43.903892    4889 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 22:30:43.905032 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:43.903950    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh" podUID="29631b37-3676-4395-a55d-f09f9b3e0025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 22:30:43.959728 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:43.959691    4889 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 22:30:43.980250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:43.959996    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t" podUID="cf853efa-c753-4ce6-834d-c315fba26177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 22:30:44.087246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:44.082082    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:44.116253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:44.105703    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 22:30:44.116253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:44.105776    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h" podUID="7cfa769f-0310-4749-a929-646c9ab9bea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 22:30:44.636459 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:44.634792    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:45.005984 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:45.005949    4889 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 22:30:45.006306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:45.006277    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh" podUID="29631b37-3676-4395-a55d-f09f9b3e0025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 22:30:45.030290 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:45.030259    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:45.031345 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:45.031197    4889 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 22:30:45.031345 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:45.031275    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t" podUID="cf853efa-c753-4ce6-834d-c315fba26177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 22:30:45.105034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:45.104572    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 22:30:45.105034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:45.104635    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h" podUID="7cfa769f-0310-4749-a929-646c9ab9bea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 22:30:45.776318 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:45.775711    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:45.928251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:45.911590    4889 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 22:30:45.928251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:45.911664    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh" podUID="29631b37-3676-4395-a55d-f09f9b3e0025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 22:30:45.962444 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:45.962400    4889 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 22:30:45.965329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:45.965292    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t" podUID="cf853efa-c753-4ce6-834d-c315fba26177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 22:30:46.103818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:46.103722    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 22:30:46.104141 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:46.104061    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h" podUID="7cfa769f-0310-4749-a929-646c9ab9bea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 22:30:46.140252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:46.136947    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:46.157697 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:46.157635    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:30:46.327597 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:46.327467    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:46.444295 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:46.441474    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6"
Oct 30 22:30:46.452469 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:46.452439    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq"
Oct 30 22:30:46.902826 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:46.902792    4889 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 22:30:46.903409 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:46.903380    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh" podUID="29631b37-3676-4395-a55d-f09f9b3e0025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 22:30:46.998534 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:46.972442    4889 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 22:30:46.998534 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:46.972547    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t" podUID="cf853efa-c753-4ce6-834d-c315fba26177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 22:30:47.104643 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:47.104322    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 22:30:47.104643 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:47.104382    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h" podUID="7cfa769f-0310-4749-a929-646c9ab9bea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 22:30:47.708657 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:47.708628    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:47.903288 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:47.903258    4889 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 22:30:47.903763 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:47.903323    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh" podUID="29631b37-3676-4395-a55d-f09f9b3e0025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 22:30:47.970432 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:47.959202    4889 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 22:30:47.970432 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:47.959278    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t" podUID="cf853efa-c753-4ce6-834d-c315fba26177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 22:30:48.106339 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:48.104961    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 22:30:48.106339 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:48.105020    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h" podUID="7cfa769f-0310-4749-a929-646c9ab9bea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 22:30:48.177041 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:48.177006    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:48.204380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:48.204346    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg"]
Oct 30 22:30:48.213521 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:48.213494    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg"
Oct 30 22:30:48.287611 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:48.270194    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-7a1ee7\""
Oct 30 22:30:48.287611 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:48.284360    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg"]
Oct 30 22:30:48.290139 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:48.290115    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w"]
Oct 30 22:30:48.295575 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:48.295435    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w"
Oct 30 22:30:48.300629 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:48.300608    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-7a1ee7-dockercfg-zlzfn\""
Oct 30 22:30:48.488593 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:48.322079    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w"]
Oct 30 22:30:48.728721 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:48.408971    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qwlc\" (UniqueName: \"kubernetes.io/projected/03c4d421-e943-45a1-91f9-71b9ec78daba-kube-api-access-8qwlc\") pod \"payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w\" (UID: \"03c4d421-e943-45a1-91f9-71b9ec78daba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w"
Oct 30 22:30:48.729070 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:48.729048    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0067cb-bba0-4ae1-9e07-3412034dac9b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg\" (UID: \"5c0067cb-bba0-4ae1-9e07-3412034dac9b\") " pod="openshift-ingress/payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg"
Oct 30 22:30:48.762615 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:48.762578    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03c4d421-e943-45a1-91f9-71b9ec78daba-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w\" (UID: \"03c4d421-e943-45a1-91f9-71b9ec78daba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w"
Oct 30 22:30:48.763172 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:48.763141    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbfm2\" (UniqueName: \"kubernetes.io/projected/5c0067cb-bba0-4ae1-9e07-3412034dac9b-kube-api-access-zbfm2\") pod \"payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg\" (UID: \"5c0067cb-bba0-4ae1-9e07-3412034dac9b\") " pod="openshift-ingress/payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg"
Oct 30 22:30:48.807469 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:48.496276    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:48.925314 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:48.916794    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbfm2\" (UniqueName: \"kubernetes.io/projected/5c0067cb-bba0-4ae1-9e07-3412034dac9b-kube-api-access-zbfm2\") pod \"payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg\" (UID: \"5c0067cb-bba0-4ae1-9e07-3412034dac9b\") " pod="openshift-ingress/payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg"
Oct 30 22:30:48.925314 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:48.916998    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qwlc\" (UniqueName: \"kubernetes.io/projected/03c4d421-e943-45a1-91f9-71b9ec78daba-kube-api-access-8qwlc\") pod \"payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w\" (UID: \"03c4d421-e943-45a1-91f9-71b9ec78daba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w"
Oct 30 22:30:48.925314 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:48.917075    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0067cb-bba0-4ae1-9e07-3412034dac9b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg\" (UID: \"5c0067cb-bba0-4ae1-9e07-3412034dac9b\") " pod="openshift-ingress/payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg"
Oct 30 22:30:48.925314 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:48.917281    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03c4d421-e943-45a1-91f9-71b9ec78daba-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w\" (UID: \"03c4d421-e943-45a1-91f9-71b9ec78daba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w"
Oct 30 22:30:48.925314 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:48.918344    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03c4d421-e943-45a1-91f9-71b9ec78daba-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w\" (UID: \"03c4d421-e943-45a1-91f9-71b9ec78daba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w"
Oct 30 22:30:48.975272 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:48.962927    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0067cb-bba0-4ae1-9e07-3412034dac9b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg\" (UID: \"5c0067cb-bba0-4ae1-9e07-3412034dac9b\") " pod="openshift-ingress/payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg"
Oct 30 22:30:49.005447 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:49.005378    4889 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 22:30:49.005447 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:49.005437    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh" podUID="29631b37-3676-4395-a55d-f09f9b3e0025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 22:30:49.086561 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:49.075778    4889 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 22:30:49.086561 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:49.075839    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t" podUID="cf853efa-c753-4ce6-834d-c315fba26177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 22:30:49.122625 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:49.117644    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbfm2\" (UniqueName: \"kubernetes.io/projected/5c0067cb-bba0-4ae1-9e07-3412034dac9b-kube-api-access-zbfm2\") pod \"payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg\" (UID: \"5c0067cb-bba0-4ae1-9e07-3412034dac9b\") " pod="openshift-ingress/payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg"
Oct 30 22:30:49.122625 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:49.119070    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qwlc\" (UniqueName: \"kubernetes.io/projected/03c4d421-e943-45a1-91f9-71b9ec78daba-kube-api-access-8qwlc\") pod \"payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w\" (UID: \"03c4d421-e943-45a1-91f9-71b9ec78daba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w"
Oct 30 22:30:49.145982 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:49.145949    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:30:49.147859 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:49.147356    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:30:49.151605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:49.151585    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 22:30:49.182645 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:49.167658    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg"
Oct 30 22:30:49.248987 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:49.248953    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:49.368269 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:49.364627    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w"
Oct 30 22:30:49.507611 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:49.507513    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:49.845462 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:30:49.845434    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c4d421_e943_45a1_91f9_71b9ec78daba.slice/crio-6d4e9617abe6b8dc8cbdf8384bde9763c578ac36ed20a9a5460c2644370c2794 WatchSource:0}: Error finding container 6d4e9617abe6b8dc8cbdf8384bde9763c578ac36ed20a9a5460c2644370c2794: Status 404 returned error can't find the container with id 6d4e9617abe6b8dc8cbdf8384bde9763c578ac36ed20a9a5460c2644370c2794
Oct 30 22:30:49.848266 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:49.847181    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w"]
Oct 30 22:30:49.853515 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:30:49.853490    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c0067cb_bba0_4ae1_9e07_3412034dac9b.slice/crio-58b5dfdaea435a9d65196308d82e6649d555d798dd2a84b71c6978ca5fa731de WatchSource:0}: Error finding container 58b5dfdaea435a9d65196308d82e6649d555d798dd2a84b71c6978ca5fa731de: Status 404 returned error can't find the container with id 58b5dfdaea435a9d65196308d82e6649d555d798dd2a84b71c6978ca5fa731de
Oct 30 22:30:49.855452 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:49.855431    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg"]
Oct 30 22:30:49.880880 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:49.880854    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:49.909429 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:49.909396    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:49.910884 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:49.910864    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:49.911599 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:49.911580    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:30:49.997682 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:49.997655    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:30:50.005051 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.005023    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:30:50.079040 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.079001    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w" event={"ID":"03c4d421-e943-45a1-91f9-71b9ec78daba","Type":"ContainerStarted","Data":"b8fef9f31a03651c323c69663c5892737e15772981e62c33e0369b589abe9c00"}
Oct 30 22:30:50.079292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.079071    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w" event={"ID":"03c4d421-e943-45a1-91f9-71b9ec78daba","Type":"ContainerStarted","Data":"6d4e9617abe6b8dc8cbdf8384bde9763c578ac36ed20a9a5460c2644370c2794"}
Oct 30 22:30:50.080083 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.080058    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w"
Oct 30 22:30:50.083623 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.083599    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg" event={"ID":"5c0067cb-bba0-4ae1-9e07-3412034dac9b","Type":"ContainerStarted","Data":"bc2b975ea14f5f2dc9df853978508c1f9fac9aa3b9a8e20d38d7546f0b1385a5"}
Oct 30 22:30:50.083795 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.083775    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg" event={"ID":"5c0067cb-bba0-4ae1-9e07-3412034dac9b","Type":"ContainerStarted","Data":"58b5dfdaea435a9d65196308d82e6649d555d798dd2a84b71c6978ca5fa731de"}
Oct 30 22:30:50.084443 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.084417    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg"
Oct 30 22:30:50.122135 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.122004    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w" podStartSLOduration=2.121986572 podStartE2EDuration="2.121986572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:50.121821961 +0000 UTC m=+1435.478188274" watchObservedRunningTime="2026-10-30 22:30:50.121986572 +0000 UTC m=+1435.478352885"
Oct 30 22:30:50.164303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.164257    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg" podStartSLOduration=2.164243344 podStartE2EDuration="2.164243344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:50.159998156 +0000 UTC m=+1435.516364467" watchObservedRunningTime="2026-10-30 22:30:50.164243344 +0000 UTC m=+1435.520609655"
Oct 30 22:30:50.328025 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.327991    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"]
Oct 30 22:30:50.333586 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.333564    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.352898 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.352875    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-3d38f6-openshift-default-dockercfg-hnzcp\""
Oct 30 22:30:50.358812 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.358786    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"]
Oct 30 22:30:50.424880 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.424765    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-envoy\") pod \"e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") " pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.424880 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.424820    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z86xp\" (UniqueName: \"kubernetes.io/projected/be263888-3047-4ee6-88cf-0fe7a033f3e1-kube-api-access-z86xp\") pod \"e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") " pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.425104 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.425004    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-podinfo\") pod \"e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") " pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.425104 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.425049    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-workload-socket\") pod \"e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") " pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.425311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.425289    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-credential-socket\") pod \"e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") " pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.425365 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.425327    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-token\") pod \"e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") " pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.425365 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.425345    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be263888-3047-4ee6-88cf-0fe7a033f3e1-istiod-ca-cert\") pod \"e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") " pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.425520 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.425503    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-workload-certs\") pod \"e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") " pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.425584 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.425569    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-data\") pod \"e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") " pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.506462 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.506139    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:50.529584 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.529542    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-podinfo\") pod \"e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") " pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.546826 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.546785    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-workload-socket\") pod \"e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") " pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.547807 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.547782    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-credential-socket\") pod \"e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") " pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.548112 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.548091    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-token\") pod \"e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") " pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.554343 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.549558    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-credential-socket\") pod \"e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") " pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.554543 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.547551    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-workload-socket\") pod \"e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") " pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.576591 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.555298    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be263888-3047-4ee6-88cf-0fe7a033f3e1-istiod-ca-cert\") pod \"e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") " pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.576892 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.554307    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be263888-3047-4ee6-88cf-0fe7a033f3e1-istiod-ca-cert\") pod \"e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") " pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.577211 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.577185    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-workload-certs\") pod \"e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") " pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.577510 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.577490    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-data\") pod \"e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") " pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.577696 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.577681    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-envoy\") pod \"e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") " pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.577838 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.577824    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z86xp\" (UniqueName: \"kubernetes.io/projected/be263888-3047-4ee6-88cf-0fe7a033f3e1-kube-api-access-z86xp\") pod \"e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") " pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.602247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.578743    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-workload-certs\") pod \"e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") " pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.617357 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.585947    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-data\") pod \"e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") " pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.617583 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.603006    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-token\") pod \"e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") " pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.617681 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.604353    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-envoy\") pod \"e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") " pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.626296 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.604798    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-podinfo\") pod \"e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") " pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.627933 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.614099    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z86xp\" (UniqueName: \"kubernetes.io/projected/be263888-3047-4ee6-88cf-0fe7a033f3e1-kube-api-access-z86xp\") pod \"e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") " pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.668705 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.666866    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:50.735882 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.735754    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:50.989645 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:50.989559    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:51.173467 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:51.173431    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:51.183453 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:30:51.183423    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe263888_3047_4ee6_88cf_0fe7a033f3e1.slice/crio-aa8bf6f74c3b715269d0cf36679cf9b887726c605f5566794d9769e31d16f68e WatchSource:0}: Error finding container aa8bf6f74c3b715269d0cf36679cf9b887726c605f5566794d9769e31d16f68e: Status 404 returned error can't find the container with id aa8bf6f74c3b715269d0cf36679cf9b887726c605f5566794d9769e31d16f68e
Oct 30 22:30:51.183920 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:51.183902    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"]
Oct 30 22:30:51.186547 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:51.186515    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:30:51.186642 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:51.186590    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:30:51.186642 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:51.186632    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:30:51.884168 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:51.884130    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:52.099870 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:52.099832    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp" event={"ID":"be263888-3047-4ee6-88cf-0fe7a033f3e1","Type":"ContainerStarted","Data":"78ef6659fd1b429d3b13edf3a54e493019dfae74033c9ecb4d421143ba19d1b9"}
Oct 30 22:30:52.100072 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:52.099915    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp" event={"ID":"be263888-3047-4ee6-88cf-0fe7a033f3e1","Type":"ContainerStarted","Data":"aa8bf6f74c3b715269d0cf36679cf9b887726c605f5566794d9769e31d16f68e"}
Oct 30 22:30:52.154288 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:52.154169    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp" podStartSLOduration=2.154156059 podStartE2EDuration="2.154156059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:52.150415269 +0000 UTC m=+1437.506781584" watchObservedRunningTime="2026-10-30 22:30:52.154156059 +0000 UTC m=+1437.510522373"
Oct 30 22:30:52.669307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:52.669274    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:52.675835 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:52.675808    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:53.113270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:53.113217    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:53.114369 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:53.114343    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:30:53.246498 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:53.246434    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:53.251362 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:53.251341    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs"]
Oct 30 22:30:53.256436 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:53.256417    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs"
Oct 30 22:30:53.270436 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:53.270413    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-d9f1c2\""
Oct 30 22:30:53.271106 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:53.271086    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-d9f1c2-dockercfg-l2tnm\""
Oct 30 22:30:53.276550 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:53.276522    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs"]
Oct 30 22:30:53.315054 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:53.315025    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk"]
Oct 30 22:30:53.320180 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:53.320155    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk"
Oct 30 22:30:53.338381 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:53.338354    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk"]
Oct 30 22:30:53.351534 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:53.351503    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94b31d20-a647-435f-8cdb-9d884c864207-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs\" (UID: \"94b31d20-a647-435f-8cdb-9d884c864207\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs"
Oct 30 22:30:53.351752 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:53.351732    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfgdp\" (UniqueName: \"kubernetes.io/projected/94b31d20-a647-435f-8cdb-9d884c864207-kube-api-access-dfgdp\") pod \"payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs\" (UID: \"94b31d20-a647-435f-8cdb-9d884c864207\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs"
Oct 30 22:30:53.542308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:53.479544    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfgdp\" (UniqueName: \"kubernetes.io/projected/94b31d20-a647-435f-8cdb-9d884c864207-kube-api-access-dfgdp\") pod \"payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs\" (UID: \"94b31d20-a647-435f-8cdb-9d884c864207\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs"
Oct 30 22:30:53.542308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:53.479607    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1511ead8-a673-47c2-9c21-9078d9e051ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk\" (UID: \"1511ead8-a673-47c2-9c21-9078d9e051ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk"
Oct 30 22:30:53.542308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:53.479880    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhs7h\" (UniqueName: \"kubernetes.io/projected/1511ead8-a673-47c2-9c21-9078d9e051ed-kube-api-access-nhs7h\") pod \"payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk\" (UID: \"1511ead8-a673-47c2-9c21-9078d9e051ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk"
Oct 30 22:30:53.542308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:53.479933    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94b31d20-a647-435f-8cdb-9d884c864207-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs\" (UID: \"94b31d20-a647-435f-8cdb-9d884c864207\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs"
Oct 30 22:30:53.542308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:53.486108    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94b31d20-a647-435f-8cdb-9d884c864207-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs\" (UID: \"94b31d20-a647-435f-8cdb-9d884c864207\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs"
Oct 30 22:30:53.542308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:53.486991    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:53.546248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:53.543908    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfgdp\" (UniqueName: \"kubernetes.io/projected/94b31d20-a647-435f-8cdb-9d884c864207-kube-api-access-dfgdp\") pod \"payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs\" (UID: \"94b31d20-a647-435f-8cdb-9d884c864207\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs"
Oct 30 22:30:53.578253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:53.574602    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs"
Oct 30 22:30:53.651256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:53.643736    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1511ead8-a673-47c2-9c21-9078d9e051ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk\" (UID: \"1511ead8-a673-47c2-9c21-9078d9e051ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk"
Oct 30 22:30:53.651256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:53.647509    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhs7h\" (UniqueName: \"kubernetes.io/projected/1511ead8-a673-47c2-9c21-9078d9e051ed-kube-api-access-nhs7h\") pod \"payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk\" (UID: \"1511ead8-a673-47c2-9c21-9078d9e051ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk"
Oct 30 22:30:53.651256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:53.650372    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1511ead8-a673-47c2-9c21-9078d9e051ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk\" (UID: \"1511ead8-a673-47c2-9c21-9078d9e051ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk"
Oct 30 22:30:53.717686 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:53.717654    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhs7h\" (UniqueName: \"kubernetes.io/projected/1511ead8-a673-47c2-9c21-9078d9e051ed-kube-api-access-nhs7h\") pod \"payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk\" (UID: \"1511ead8-a673-47c2-9c21-9078d9e051ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk"
Oct 30 22:30:53.945410 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:53.945381    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk"
Oct 30 22:30:54.049622 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:54.049537    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs"]
Oct 30 22:30:54.067248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:54.067057    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:54.266992 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:30:54.266854    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1511ead8_a673_47c2_9c21_9078d9e051ed.slice/crio-20bd03ff9056dec08b225b7f4df97b0d8c5eddd1c812156231303ce00c17a0a9 WatchSource:0}: Error finding container 20bd03ff9056dec08b225b7f4df97b0d8c5eddd1c812156231303ce00c17a0a9: Status 404 returned error can't find the container with id 20bd03ff9056dec08b225b7f4df97b0d8c5eddd1c812156231303ce00c17a0a9
Oct 30 22:30:54.268310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:54.268285    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk"]
Oct 30 22:30:54.574897 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:54.574826    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:54.747938 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:54.747905    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:55.180016 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:55.179977    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs" event={"ID":"94b31d20-a647-435f-8cdb-9d884c864207","Type":"ContainerStarted","Data":"5f48b4eaa5b692860fc67f9265ef6cad9934cac8c3ad78583111a9c62cb38a51"}
Oct 30 22:30:55.180211 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:55.180025    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs" event={"ID":"94b31d20-a647-435f-8cdb-9d884c864207","Type":"ContainerStarted","Data":"ca05f80e583050c5fbb62daf6a61f14178a046fcd3a790b60792640e4cebcba1"}
Oct 30 22:30:55.180211 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:55.180085    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs"
Oct 30 22:30:55.182102 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:55.182071    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk" event={"ID":"1511ead8-a673-47c2-9c21-9078d9e051ed","Type":"ContainerStarted","Data":"79e7d3f65230f8769b2887a5af8e8c347bf1ebbb5b73f0b1139ac30f24ed4f9c"}
Oct 30 22:30:55.182214 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:55.182109    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk" event={"ID":"1511ead8-a673-47c2-9c21-9078d9e051ed","Type":"ContainerStarted","Data":"20bd03ff9056dec08b225b7f4df97b0d8c5eddd1c812156231303ce00c17a0a9"}
Oct 30 22:30:55.182276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:55.182253    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk"
Oct 30 22:30:55.233003 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:55.232943    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs" podStartSLOduration=2.232929199 podStartE2EDuration="2.232929199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:55.23151449 +0000 UTC m=+1440.587880805" watchObservedRunningTime="2026-10-30 22:30:55.232929199 +0000 UTC m=+1440.589295513"
Oct 30 22:30:55.287174 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:55.287118    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk" podStartSLOduration=2.287105194 podStartE2EDuration="2.287105194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:55.28554185 +0000 UTC m=+1440.641908163" watchObservedRunningTime="2026-10-30 22:30:55.287105194 +0000 UTC m=+1440.643471539"
Oct 30 22:30:56.038247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:56.031313    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:56.273410 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:56.273353    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:56.454299 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:56.450290    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:57.938355 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:57.937271    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:58.361824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:58.361795    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:59.691623 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:30:59.691578    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:01.457138 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:01.457098    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:02.103349 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:02.103317    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w"
Oct 30 22:31:02.103768 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:02.103750    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg"
Oct 30 22:31:03.179451 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:03.179402    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:04.953582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:04.953548    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:06.200417 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:06.200393    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs"
Oct 30 22:31:06.200824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:06.200457    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk"
Oct 30 22:31:06.660740 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:06.660706    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:06.825108 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:06.825076    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:08.447637 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:08.447603    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:09.000691 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:09.000659    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn"]
Oct 30 22:31:09.505943 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:09.505877    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"]
Oct 30 22:31:09.507394 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:09.507343    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w" podUID="b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5" containerName="istio-proxy" containerID="cri-o://4b1f9d8a2197c754b46f0d96fe1295bc6f9040b6db73bd6ae1038e5a3d4ba651" gracePeriod=30
Oct 30 22:31:09.540022 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:09.539978    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"]
Oct 30 22:31:09.547165 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:09.547129    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7" podUID="d15e2f68-d995-43a9-aa5e-32aec47af6f7" containerName="istio-proxy" containerID="cri-o://6aeac9d62246834e8a096dc043515debab44db6b8421e554ea1e31ef13f60a54" gracePeriod=30
Oct 30 22:31:10.194075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:10.193043    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"]
Oct 30 22:31:10.194075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:10.193344    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt" podUID="3084be58-a1e9-410a-b501-2c71cca33742" containerName="istio-proxy" containerID="cri-o://0acf6cf2a7c17874651cd2ea4bdb41059fc147d1ecf885d2b013ff68be84a870" gracePeriod=30
Oct 30 22:31:10.339503 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:10.334424    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"]
Oct 30 22:31:10.339503 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:10.334753    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt" podUID="fe40461b-fff1-4d46-84d5-262b608028b2" containerName="istio-proxy" containerID="cri-o://faef3ba24dd7105bccc2015733b748e403eacf8270ff455ccfc4a9521402e9a5" gracePeriod=30
Oct 30 22:31:10.384721 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:31:10.384683    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:31:10.777838 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:10.774980    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"]
Oct 30 22:31:10.805433 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:10.805400    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:10.824096 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:10.824068    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"]
Oct 30 22:31:10.902944 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:10.902897    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:10.903132 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:10.902955    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:10.903132 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:10.902990    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:10.903132 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:10.903053    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5fkc\" (UniqueName: \"kubernetes.io/projected/03297afe-0b96-48ea-bcb8-849f2cd95bb1-kube-api-access-h5fkc\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:10.903336 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:10.903312    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:10.903590 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:10.903562    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:10.903694 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:10.903610    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:10.903754 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:10.903720    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:10.903836 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:10.903820    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:11.005730 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:11.005699    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:11.005915 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:11.005809    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:11.005915 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:11.005836    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:11.005915 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:11.005860    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:11.005915 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:11.005877    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5fkc\" (UniqueName: \"kubernetes.io/projected/03297afe-0b96-48ea-bcb8-849f2cd95bb1-kube-api-access-h5fkc\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:11.006134 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:11.005990    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:11.006134 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:11.006074    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:11.006134 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:11.006101    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:11.006134 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:11.006154    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:11.006134 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:11.006310    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:11.006727 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:11.006516    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:11.006727 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:11.006564    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:11.006727 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:11.006650    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:11.006727 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:11.006688    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:11.008441 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:11.008421    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:11.008811 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:11.008795    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:11.039594 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:11.039518    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:11.039720 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:11.039653    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5fkc\" (UniqueName: \"kubernetes.io/projected/03297afe-0b96-48ea-bcb8-849f2cd95bb1-kube-api-access-h5fkc\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:11.134258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:11.132571    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:11.391509 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:11.391421    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:11.463063 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:31:11.463028    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03297afe_0b96_48ea_bcb8_849f2cd95bb1.slice/crio-26a3fe67c5e8df9d3a7c093afe6d9458bd0e958354074df5e879d7e7e2afdd23 WatchSource:0}: Error finding container 26a3fe67c5e8df9d3a7c093afe6d9458bd0e958354074df5e879d7e7e2afdd23: Status 404 returned error can't find the container with id 26a3fe67c5e8df9d3a7c093afe6d9458bd0e958354074df5e879d7e7e2afdd23
Oct 30 22:31:11.464622 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:11.464594    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"]
Oct 30 22:31:11.466319 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:11.466286    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:31:11.466392 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:11.466353    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:31:11.466430 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:11.466388    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:31:11.565656 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:11.565631    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:12.379492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:12.379460    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95" event={"ID":"03297afe-0b96-48ea-bcb8-849f2cd95bb1","Type":"ContainerStarted","Data":"09ff567bd01776e200fe9ec7272caa5ef2af745275e006d00cc853fd1206c436"}
Oct 30 22:31:12.379866 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:12.379509    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95" event={"ID":"03297afe-0b96-48ea-bcb8-849f2cd95bb1","Type":"ContainerStarted","Data":"26a3fe67c5e8df9d3a7c093afe6d9458bd0e958354074df5e879d7e7e2afdd23"}
Oct 30 22:31:12.462160 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:12.460999    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95" podStartSLOduration=2.460977471 podStartE2EDuration="2.460977471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:12.443719776 +0000 UTC m=+1457.800086091" watchObservedRunningTime="2026-10-30 22:31:12.460977471 +0000 UTC m=+1457.817343811"
Oct 30 22:31:12.570251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:12.558306    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:13.139334 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:13.139288    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:13.145399 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:13.145365    4889 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 22:31:13.145686 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:13.145655    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95" podUID="03297afe-0b96-48ea-bcb8-849f2cd95bb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 22:31:14.001249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:14.001189    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn" podUID="440bb07c-dd47-45ed-a095-f4247e739d36" containerName="payload-processing" containerID="cri-o://28bda0e483290039c099bb818b4514acae755c6abfaa7fbfd47c318994afa147" gracePeriod=25
Oct 30 22:31:14.142251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:14.139765    4889 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 22:31:14.142251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:14.139842    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95" podUID="03297afe-0b96-48ea-bcb8-849f2cd95bb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 22:31:14.775874 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:14.745496    4889 generic.go:356] "Generic (PLEG): container finished" podID="440bb07c-dd47-45ed-a095-f4247e739d36" containerID="28bda0e483290039c099bb818b4514acae755c6abfaa7fbfd47c318994afa147" exitCode=0
Oct 30 22:31:14.775874 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:14.750271    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn" event={"ID":"440bb07c-dd47-45ed-a095-f4247e739d36","Type":"ContainerDied","Data":"28bda0e483290039c099bb818b4514acae755c6abfaa7fbfd47c318994afa147"}
Oct 30 22:31:15.177664 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.177008    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:15.198334 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.198282    4889 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 22:31:15.198591 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.198552    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95" podUID="03297afe-0b96-48ea-bcb8-849f2cd95bb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 22:31:15.439518 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.421785    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:15.475258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.474783    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn"
Oct 30 22:31:15.507072 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.507024    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/440bb07c-dd47-45ed-a095-f4247e739d36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440bb07c-dd47-45ed-a095-f4247e739d36-plugins-config-volume\") pod \"440bb07c-dd47-45ed-a095-f4247e739d36\" (UID: \"440bb07c-dd47-45ed-a095-f4247e739d36\") "
Oct 30 22:31:15.508082 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.508052    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440bb07c-dd47-45ed-a095-f4247e739d36-plugins-config-volume" pod "440bb07c-dd47-45ed-a095-f4247e739d36" (UID: "440bb07c-dd47-45ed-a095-f4247e739d36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:15.508808 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.508785    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/440bb07c-dd47-45ed-a095-f4247e739d36-kube-api-access-mqxhr\" (UniqueName: \"kubernetes.io/projected/440bb07c-dd47-45ed-a095-f4247e739d36-kube-api-access-mqxhr\") pod \"440bb07c-dd47-45ed-a095-f4247e739d36\" (UID: \"440bb07c-dd47-45ed-a095-f4247e739d36\") "
Oct 30 22:31:15.577479 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.577123    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440bb07c-dd47-45ed-a095-f4247e739d36-kube-api-access-mqxhr" pod "440bb07c-dd47-45ed-a095-f4247e739d36" (UID: "440bb07c-dd47-45ed-a095-f4247e739d36"). InnerVolumeSpecName "kube-api-access-mqxhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:15.582352 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.582309    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqxhr\" (UniqueName: \"kubernetes.io/projected/440bb07c-dd47-45ed-a095-f4247e739d36-kube-api-access-mqxhr\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.582352 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.582346    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440bb07c-dd47-45ed-a095-f4247e739d36-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.604400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.599850    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:31:15.787545 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.787511    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istiod-ca-cert\") pod \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") "
Oct 30 22:31:15.788015 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.787993    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-credential-socket\") pod \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") "
Oct 30 22:31:15.788157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.788031    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-workload-socket\") pod \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") "
Oct 30 22:31:15.788157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.788109    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-workload-certs\") pod \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") "
Oct 30 22:31:15.788157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.788148    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-envoy\") pod \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") "
Oct 30 22:31:15.788347 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.788198    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-token\" (UniqueName: \"kubernetes.io/projected/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-token\") pod \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") "
Oct 30 22:31:15.788347 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.788291    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-kube-api-access-jqdb9\" (UniqueName: \"kubernetes.io/projected/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-kube-api-access-jqdb9\") pod \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") "
Oct 30 22:31:15.788455 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.788352    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-data\") pod \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") "
Oct 30 22:31:15.788455 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.788414    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-podinfo\") pod \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\" (UID: \"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5\") "
Oct 30 22:31:15.789202 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.789172    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-workload-certs" pod "b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5" (UID: "b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:15.789485 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.789457    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-credential-socket" pod "b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5" (UID: "b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:15.789656 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.789634    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-workload-socket" pod "b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5" (UID: "b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:15.790152 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.790127    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-data" pod "b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5" (UID: "b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:15.790268 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.790251    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istiod-ca-cert" pod "b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5" (UID: "b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:15.791996 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.791964    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-token" pod "b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5" (UID: "b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:15.793242 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.793190    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-envoy" pod "b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5" (UID: "b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:15.793465 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.793440    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-kube-api-access-jqdb9" pod "b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5" (UID: "b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5"). InnerVolumeSpecName "kube-api-access-jqdb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:15.794460 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.794424    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-podinfo" pod "b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5" (UID: "b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:31:15.801513 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.801488    4889 generic.go:356] "Generic (PLEG): container finished" podID="d15e2f68-d995-43a9-aa5e-32aec47af6f7" containerID="6aeac9d62246834e8a096dc043515debab44db6b8421e554ea1e31ef13f60a54" exitCode=0
Oct 30 22:31:15.801721 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.801702    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7" event={"ID":"d15e2f68-d995-43a9-aa5e-32aec47af6f7","Type":"ContainerDied","Data":"6aeac9d62246834e8a096dc043515debab44db6b8421e554ea1e31ef13f60a54"}
Oct 30 22:31:15.804116 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.804103    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn"
Oct 30 22:31:15.804418 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.804218    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn" event={"ID":"440bb07c-dd47-45ed-a095-f4247e739d36","Type":"ContainerDied","Data":"c86af3ff6a3e6f5ea0943e99a844ead328f00c1463239c6a0088b706aa2c4a7d"}
Oct 30 22:31:15.804418 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.804307    4889 scope.go:122] "RemoveContainer" containerID="28bda0e483290039c099bb818b4514acae755c6abfaa7fbfd47c318994afa147"
Oct 30 22:31:15.826534 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.809361    4889 generic.go:356] "Generic (PLEG): container finished" podID="3084be58-a1e9-410a-b501-2c71cca33742" containerID="0acf6cf2a7c17874651cd2ea4bdb41059fc147d1ecf885d2b013ff68be84a870" exitCode=0
Oct 30 22:31:15.826534 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.809521    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt" event={"ID":"3084be58-a1e9-410a-b501-2c71cca33742","Type":"ContainerDied","Data":"0acf6cf2a7c17874651cd2ea4bdb41059fc147d1ecf885d2b013ff68be84a870"}
Oct 30 22:31:15.878373 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.878129    4889 generic.go:356] "Generic (PLEG): container finished" podID="fe40461b-fff1-4d46-84d5-262b608028b2" containerID="faef3ba24dd7105bccc2015733b748e403eacf8270ff455ccfc4a9521402e9a5" exitCode=0
Oct 30 22:31:15.878373 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.878305    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt" event={"ID":"fe40461b-fff1-4d46-84d5-262b608028b2","Type":"ContainerDied","Data":"faef3ba24dd7105bccc2015733b748e403eacf8270ff455ccfc4a9521402e9a5"}
Oct 30 22:31:15.881398 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.881371    4889 generic.go:356] "Generic (PLEG): container finished" podID="b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5" containerID="4b1f9d8a2197c754b46f0d96fe1295bc6f9040b6db73bd6ae1038e5a3d4ba651" exitCode=0
Oct 30 22:31:15.881520 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.881436    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w" event={"ID":"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5","Type":"ContainerDied","Data":"4b1f9d8a2197c754b46f0d96fe1295bc6f9040b6db73bd6ae1038e5a3d4ba651"}
Oct 30 22:31:15.881520 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.881477    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w" event={"ID":"b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5","Type":"ContainerDied","Data":"c6d0090bdf599acba2d28d834982bf7525754780395f2cc78dde03e1375e97da"}
Oct 30 22:31:15.881641 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.881555    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"
Oct 30 22:31:15.891061 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.890915    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.891061 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.890943    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqdb9\" (UniqueName: \"kubernetes.io/projected/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-kube-api-access-jqdb9\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.891061 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.890959    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.891061 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.890973    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.891061 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.890986    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.891061 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.890999    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.891061 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.891012    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.891061 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.891026    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.891061 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.891039    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.893024 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.892799    4889 scope.go:122] "RemoveContainer" containerID="4b1f9d8a2197c754b46f0d96fe1295bc6f9040b6db73bd6ae1038e5a3d4ba651"
Oct 30 22:31:15.933848 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.933331    4889 scope.go:122] "RemoveContainer" containerID="4b1f9d8a2197c754b46f0d96fe1295bc6f9040b6db73bd6ae1038e5a3d4ba651"
Oct 30 22:31:15.962390 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:31:15.953741    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b1f9d8a2197c754b46f0d96fe1295bc6f9040b6db73bd6ae1038e5a3d4ba651\": container with ID starting with 4b1f9d8a2197c754b46f0d96fe1295bc6f9040b6db73bd6ae1038e5a3d4ba651 not found: ID does not exist" containerID="4b1f9d8a2197c754b46f0d96fe1295bc6f9040b6db73bd6ae1038e5a3d4ba651"
Oct 30 22:31:15.962390 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.953780    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b1f9d8a2197c754b46f0d96fe1295bc6f9040b6db73bd6ae1038e5a3d4ba651"} err="failed to get container status \"4b1f9d8a2197c754b46f0d96fe1295bc6f9040b6db73bd6ae1038e5a3d4ba651\": rpc error: code = NotFound desc = could not find container \"4b1f9d8a2197c754b46f0d96fe1295bc6f9040b6db73bd6ae1038e5a3d4ba651\": container with ID starting with 4b1f9d8a2197c754b46f0d96fe1295bc6f9040b6db73bd6ae1038e5a3d4ba651 not found: ID does not exist"
Oct 30 22:31:15.971954 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.971919    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn"]
Oct 30 22:31:15.982981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.982938    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2949c143-5bdffff8fc-jk5zn"]
Oct 30 22:31:15.984814 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.984788    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:31:15.988497 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.988477    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"]
Oct 30 22:31:15.991897 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.991872    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-envoy\") pod \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") "
Oct 30 22:31:15.992006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.991994    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-workload-certs\") pod \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") "
Oct 30 22:31:15.992063 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.992054    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-data\") pod \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") "
Oct 30 22:31:15.992133 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.992118    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-podinfo\") pod \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") "
Oct 30 22:31:15.992194 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.992167    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d15e2f68-d995-43a9-aa5e-32aec47af6f7-kube-api-access-z4qff\" (UniqueName: \"kubernetes.io/projected/d15e2f68-d995-43a9-aa5e-32aec47af6f7-kube-api-access-z4qff\") pod \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") "
Oct 30 22:31:15.992300 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.992205    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istiod-ca-cert\") pod \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") "
Oct 30 22:31:15.992371 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.992321    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-credential-socket\") pod \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") "
Oct 30 22:31:15.992371 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.992359    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-token\" (UniqueName: \"kubernetes.io/projected/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-token\") pod \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") "
Oct 30 22:31:15.992470 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.992398    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-workload-socket\") pod \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\" (UID: \"d15e2f68-d995-43a9-aa5e-32aec47af6f7\") "
Oct 30 22:31:15.993559 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.993373    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-credential-socket" pod "d15e2f68-d995-43a9-aa5e-32aec47af6f7" (UID: "d15e2f68-d995-43a9-aa5e-32aec47af6f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:15.993559 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.993403    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-workload-socket" pod "d15e2f68-d995-43a9-aa5e-32aec47af6f7" (UID: "d15e2f68-d995-43a9-aa5e-32aec47af6f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:15.993707 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.993594    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-data" pod "d15e2f68-d995-43a9-aa5e-32aec47af6f7" (UID: "d15e2f68-d995-43a9-aa5e-32aec47af6f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:15.993918 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.993804    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-8r62w"]
Oct 30 22:31:15.993918 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.993883    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istiod-ca-cert" pod "d15e2f68-d995-43a9-aa5e-32aec47af6f7" (UID: "d15e2f68-d995-43a9-aa5e-32aec47af6f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:15.994338 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.994307    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-workload-certs" pod "d15e2f68-d995-43a9-aa5e-32aec47af6f7" (UID: "d15e2f68-d995-43a9-aa5e-32aec47af6f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:15.995771 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.995726    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-envoy" pod "d15e2f68-d995-43a9-aa5e-32aec47af6f7" (UID: "d15e2f68-d995-43a9-aa5e-32aec47af6f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:15.995878 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:15.995826    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d15e2f68-d995-43a9-aa5e-32aec47af6f7-kube-api-access-z4qff" pod "d15e2f68-d995-43a9-aa5e-32aec47af6f7" (UID: "d15e2f68-d995-43a9-aa5e-32aec47af6f7"). InnerVolumeSpecName "kube-api-access-z4qff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:16.002165 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.002141    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-podinfo" pod "d15e2f68-d995-43a9-aa5e-32aec47af6f7" (UID: "d15e2f68-d995-43a9-aa5e-32aec47af6f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:31:16.004097 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.004063    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-token" pod "d15e2f68-d995-43a9-aa5e-32aec47af6f7" (UID: "d15e2f68-d995-43a9-aa5e-32aec47af6f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:16.124180 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.123979    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.124180 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.124028    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.124180 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.124043    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.124180 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.124059    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4qff\" (UniqueName: \"kubernetes.io/projected/d15e2f68-d995-43a9-aa5e-32aec47af6f7-kube-api-access-z4qff\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.124180 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.124077    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.124180 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.124096    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.124180 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.124112    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.124180 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.124128    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.124180 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.124142    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d15e2f68-d995-43a9-aa5e-32aec47af6f7-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.176492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.175484    4889 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 22:31:16.176492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.175548    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95" podUID="03297afe-0b96-48ea-bcb8-849f2cd95bb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 22:31:16.393462 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.393441    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:31:16.395443 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.395425    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:31:16.587304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.475921    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-istio-envoy\") pod \"fe40461b-fff1-4d46-84d5-262b608028b2\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") "
Oct 30 22:31:16.587304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.475983    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe40461b-fff1-4d46-84d5-262b608028b2-kube-api-access-sq4rn\" (UniqueName: \"kubernetes.io/projected/fe40461b-fff1-4d46-84d5-262b608028b2-kube-api-access-sq4rn\") pod \"fe40461b-fff1-4d46-84d5-262b608028b2\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") "
Oct 30 22:31:16.587304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.476082    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3084be58-a1e9-410a-b501-2c71cca33742-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3084be58-a1e9-410a-b501-2c71cca33742-istiod-ca-cert\") pod \"3084be58-a1e9-410a-b501-2c71cca33742\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") "
Oct 30 22:31:16.587304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.476130    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-istio-data\") pod \"3084be58-a1e9-410a-b501-2c71cca33742\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") "
Oct 30 22:31:16.587304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.476185    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-istio-envoy\") pod \"3084be58-a1e9-410a-b501-2c71cca33742\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") "
Oct 30 22:31:16.587304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.476273    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-istio-data\") pod \"fe40461b-fff1-4d46-84d5-262b608028b2\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") "
Oct 30 22:31:16.587304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.476309    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe40461b-fff1-4d46-84d5-262b608028b2-istio-token\" (UniqueName: \"kubernetes.io/projected/fe40461b-fff1-4d46-84d5-262b608028b2-istio-token\") pod \"fe40461b-fff1-4d46-84d5-262b608028b2\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") "
Oct 30 22:31:16.587304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.476344    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-workload-socket\") pod \"fe40461b-fff1-4d46-84d5-262b608028b2\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") "
Oct 30 22:31:16.587304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.476380    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3084be58-a1e9-410a-b501-2c71cca33742-kube-api-access-fjfhn\" (UniqueName: \"kubernetes.io/projected/3084be58-a1e9-410a-b501-2c71cca33742-kube-api-access-fjfhn\") pod \"3084be58-a1e9-410a-b501-2c71cca33742\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") "
Oct 30 22:31:16.587304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.476408    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe40461b-fff1-4d46-84d5-262b608028b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe40461b-fff1-4d46-84d5-262b608028b2-istio-podinfo\") pod \"fe40461b-fff1-4d46-84d5-262b608028b2\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") "
Oct 30 22:31:16.587304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.476480    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3084be58-a1e9-410a-b501-2c71cca33742-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3084be58-a1e9-410a-b501-2c71cca33742-istio-podinfo\") pod \"3084be58-a1e9-410a-b501-2c71cca33742\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") "
Oct 30 22:31:16.587304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.477018    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-workload-socket" pod "fe40461b-fff1-4d46-84d5-262b608028b2" (UID: "fe40461b-fff1-4d46-84d5-262b608028b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:16.587304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.477292    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-istio-data" pod "fe40461b-fff1-4d46-84d5-262b608028b2" (UID: "fe40461b-fff1-4d46-84d5-262b608028b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:16.587304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.477868    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3084be58-a1e9-410a-b501-2c71cca33742-istiod-ca-cert" pod "3084be58-a1e9-410a-b501-2c71cca33742" (UID: "3084be58-a1e9-410a-b501-2c71cca33742"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:16.587304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.518555    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-istio-envoy" pod "3084be58-a1e9-410a-b501-2c71cca33742" (UID: "3084be58-a1e9-410a-b501-2c71cca33742"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:16.587304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.522389    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-istio-envoy" pod "fe40461b-fff1-4d46-84d5-262b608028b2" (UID: "fe40461b-fff1-4d46-84d5-262b608028b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:16.587304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.524358    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-istio-data" pod "3084be58-a1e9-410a-b501-2c71cca33742" (UID: "3084be58-a1e9-410a-b501-2c71cca33742"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:16.587304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.539432    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe40461b-fff1-4d46-84d5-262b608028b2-istio-token" pod "fe40461b-fff1-4d46-84d5-262b608028b2" (UID: "fe40461b-fff1-4d46-84d5-262b608028b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:16.587304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.539529    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe40461b-fff1-4d46-84d5-262b608028b2-istio-podinfo" pod "fe40461b-fff1-4d46-84d5-262b608028b2" (UID: "fe40461b-fff1-4d46-84d5-262b608028b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:31:16.587304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.539588    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3084be58-a1e9-410a-b501-2c71cca33742-kube-api-access-fjfhn" pod "3084be58-a1e9-410a-b501-2c71cca33742" (UID: "3084be58-a1e9-410a-b501-2c71cca33742"). InnerVolumeSpecName "kube-api-access-fjfhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:16.587304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.540554    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe40461b-fff1-4d46-84d5-262b608028b2-kube-api-access-sq4rn" pod "fe40461b-fff1-4d46-84d5-262b608028b2" (UID: "fe40461b-fff1-4d46-84d5-262b608028b2"). InnerVolumeSpecName "kube-api-access-sq4rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:16.587304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.557967    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3084be58-a1e9-410a-b501-2c71cca33742-istio-podinfo" pod "3084be58-a1e9-410a-b501-2c71cca33742" (UID: "3084be58-a1e9-410a-b501-2c71cca33742"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:31:16.622946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.621352    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-workload-certs\") pod \"3084be58-a1e9-410a-b501-2c71cca33742\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") "
Oct 30 22:31:16.622946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.621407    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3084be58-a1e9-410a-b501-2c71cca33742-istio-token\" (UniqueName: \"kubernetes.io/projected/3084be58-a1e9-410a-b501-2c71cca33742-istio-token\") pod \"3084be58-a1e9-410a-b501-2c71cca33742\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") "
Oct 30 22:31:16.622946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.621444    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-credential-socket\") pod \"fe40461b-fff1-4d46-84d5-262b608028b2\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") "
Oct 30 22:31:16.622946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.621488    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe40461b-fff1-4d46-84d5-262b608028b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe40461b-fff1-4d46-84d5-262b608028b2-istiod-ca-cert\") pod \"fe40461b-fff1-4d46-84d5-262b608028b2\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") "
Oct 30 22:31:16.622946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.621520    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-workload-socket\") pod \"3084be58-a1e9-410a-b501-2c71cca33742\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") "
Oct 30 22:31:16.622946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.621582    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-credential-socket\") pod \"3084be58-a1e9-410a-b501-2c71cca33742\" (UID: \"3084be58-a1e9-410a-b501-2c71cca33742\") "
Oct 30 22:31:16.622946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.621624    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-workload-certs\") pod \"fe40461b-fff1-4d46-84d5-262b608028b2\" (UID: \"fe40461b-fff1-4d46-84d5-262b608028b2\") "
Oct 30 22:31:16.622946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.622559    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3084be58-a1e9-410a-b501-2c71cca33742-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.622946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.622585    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.622946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.622599    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.622946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.622612    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.622946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.622626    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe40461b-fff1-4d46-84d5-262b608028b2-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.622946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.622638    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.622946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.622652    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjfhn\" (UniqueName: \"kubernetes.io/projected/3084be58-a1e9-410a-b501-2c71cca33742-kube-api-access-fjfhn\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.622946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.622666    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe40461b-fff1-4d46-84d5-262b608028b2-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.622946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.622678    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3084be58-a1e9-410a-b501-2c71cca33742-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.622946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.622691    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.622946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.622707    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq4rn\" (UniqueName: \"kubernetes.io/projected/fe40461b-fff1-4d46-84d5-262b608028b2-kube-api-access-sq4rn\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.642211 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.624680    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-workload-socket" pod "3084be58-a1e9-410a-b501-2c71cca33742" (UID: "3084be58-a1e9-410a-b501-2c71cca33742"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:16.642211 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.627562    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-workload-certs" pod "3084be58-a1e9-410a-b501-2c71cca33742" (UID: "3084be58-a1e9-410a-b501-2c71cca33742"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:16.642211 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.641328    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-credential-socket" pod "3084be58-a1e9-410a-b501-2c71cca33742" (UID: "3084be58-a1e9-410a-b501-2c71cca33742"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:16.642211 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.641328    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-credential-socket" pod "fe40461b-fff1-4d46-84d5-262b608028b2" (UID: "fe40461b-fff1-4d46-84d5-262b608028b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:16.642465 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.642423    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe40461b-fff1-4d46-84d5-262b608028b2-istiod-ca-cert" pod "fe40461b-fff1-4d46-84d5-262b608028b2" (UID: "fe40461b-fff1-4d46-84d5-262b608028b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:16.642646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.642603    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-workload-certs" pod "fe40461b-fff1-4d46-84d5-262b608028b2" (UID: "fe40461b-fff1-4d46-84d5-262b608028b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:16.654473 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.654412    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3084be58-a1e9-410a-b501-2c71cca33742-istio-token" pod "3084be58-a1e9-410a-b501-2c71cca33742" (UID: "3084be58-a1e9-410a-b501-2c71cca33742"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:16.779115 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.734731    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.779115 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.734774    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3084be58-a1e9-410a-b501-2c71cca33742-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.779115 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.734788    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.779115 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.734803    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe40461b-fff1-4d46-84d5-262b608028b2-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.779115 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.734816    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.779115 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.734829    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3084be58-a1e9-410a-b501-2c71cca33742-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.779115 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.734842    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe40461b-fff1-4d46-84d5-262b608028b2-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.870939 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.870856    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="440bb07c-dd47-45ed-a095-f4247e739d36" path="/var/lib/kubelet/pods/440bb07c-dd47-45ed-a095-f4247e739d36/volumes"
Oct 30 22:31:16.871927 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.871884    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5" path="/var/lib/kubelet/pods/b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5/volumes"
Oct 30 22:31:16.893603 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.893577    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt" event={"ID":"fe40461b-fff1-4d46-84d5-262b608028b2","Type":"ContainerDied","Data":"e3b5bc6136144b4dcf18668f8400113d9a81d0d7aeb9bcd28455d02965fcf17d"}
Oct 30 22:31:16.893703 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.893634    4889 scope.go:122] "RemoveContainer" containerID="faef3ba24dd7105bccc2015733b748e403eacf8270ff455ccfc4a9521402e9a5"
Oct 30 22:31:16.893739 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.893718    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"
Oct 30 22:31:16.898347 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.898184    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7" event={"ID":"d15e2f68-d995-43a9-aa5e-32aec47af6f7","Type":"ContainerDied","Data":"2f7cc7d3a64200486cd54b086f307e48032d05ee69f09339491b0512f2f1a1b0"}
Oct 30 22:31:16.899058 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.899036    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"
Oct 30 22:31:16.904471 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.901759    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt" event={"ID":"3084be58-a1e9-410a-b501-2c71cca33742","Type":"ContainerDied","Data":"0873359859d82861b7a9fe0680d480e8cb5b502f6efb2fc872234663f64538a4"}
Oct 30 22:31:16.904471 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.901807    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"
Oct 30 22:31:16.944651 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.944614    4889 scope.go:122] "RemoveContainer" containerID="6aeac9d62246834e8a096dc043515debab44db6b8421e554ea1e31ef13f60a54"
Oct 30 22:31:16.957301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.957278    4889 scope.go:122] "RemoveContainer" containerID="0acf6cf2a7c17874651cd2ea4bdb41059fc147d1ecf885d2b013ff68be84a870"
Oct 30 22:31:16.975789 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.975761    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"]
Oct 30 22:31:16.985894 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:16.985864    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-nflrt"]
Oct 30 22:31:17.046675 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:17.045527    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"]
Oct 30 22:31:17.050207 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:17.050185    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8wzt7"]
Oct 30 22:31:17.072324 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:17.072297    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"]
Oct 30 22:31:17.082368 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:17.082345    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-58wkt"]
Oct 30 22:31:17.137168 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:17.136657    4889 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 22:31:17.137168 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:17.136722    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95" podUID="03297afe-0b96-48ea-bcb8-849f2cd95bb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 22:31:17.755253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:17.751793    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:18.135303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:18.134686    4889 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 22:31:18.135303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:18.134747    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95" podUID="03297afe-0b96-48ea-bcb8-849f2cd95bb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 22:31:18.777063 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:18.777021    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3084be58-a1e9-410a-b501-2c71cca33742" path="/var/lib/kubelet/pods/3084be58-a1e9-410a-b501-2c71cca33742/volumes"
Oct 30 22:31:18.777803 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:18.777782    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d15e2f68-d995-43a9-aa5e-32aec47af6f7" path="/var/lib/kubelet/pods/d15e2f68-d995-43a9-aa5e-32aec47af6f7/volumes"
Oct 30 22:31:18.778333 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:18.778312    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe40461b-fff1-4d46-84d5-262b608028b2" path="/var/lib/kubelet/pods/fe40461b-fff1-4d46-84d5-262b608028b2/volumes"
Oct 30 22:31:19.139399 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:19.139327    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:19.139889 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:19.139866    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:19.140787 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:19.140764    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:31:19.794246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:19.785183    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:20.156800 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:20.153525    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:20.775675 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:20.775625    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:21.143966 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.143776    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"]
Oct 30 22:31:21.145154 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.145128    4889 cpu_manager.go:414] "containerMap: removing container" podUID="fe40461b-fff1-4d46-84d5-262b608028b2" containerName="istio-proxy"
Oct 30 22:31:21.145274 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.145158    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe40461b-fff1-4d46-84d5-262b608028b2" containerName="istio-proxy"
Oct 30 22:31:21.145341 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.145294    4889 cpu_manager.go:414] "containerMap: removing container" podUID="d15e2f68-d995-43a9-aa5e-32aec47af6f7" containerName="istio-proxy"
Oct 30 22:31:21.145341 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.145306    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d15e2f68-d995-43a9-aa5e-32aec47af6f7" containerName="istio-proxy"
Oct 30 22:31:21.145341 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.145326    4889 cpu_manager.go:414] "containerMap: removing container" podUID="3084be58-a1e9-410a-b501-2c71cca33742" containerName="istio-proxy"
Oct 30 22:31:21.145341 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.145332    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3084be58-a1e9-410a-b501-2c71cca33742" containerName="istio-proxy"
Oct 30 22:31:21.145341 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.145339    4889 cpu_manager.go:414] "containerMap: removing container" podUID="440bb07c-dd47-45ed-a095-f4247e739d36" containerName="payload-processing"
Oct 30 22:31:21.145341 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.145344    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440bb07c-dd47-45ed-a095-f4247e739d36" containerName="payload-processing"
Oct 30 22:31:21.145636 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.145353    4889 cpu_manager.go:414] "containerMap: removing container" podUID="b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5" containerName="istio-proxy"
Oct 30 22:31:21.145636 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.145358    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5" containerName="istio-proxy"
Oct 30 22:31:21.145636 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.145545    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="d15e2f68-d995-43a9-aa5e-32aec47af6f7" containerName="istio-proxy"
Oct 30 22:31:21.145636 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.145561    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="3084be58-a1e9-410a-b501-2c71cca33742" containerName="istio-proxy"
Oct 30 22:31:21.145636 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.145571    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="440bb07c-dd47-45ed-a095-f4247e739d36" containerName="payload-processing"
Oct 30 22:31:21.145636 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.145582    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0770be9-8ba4-4b85-8c96-6e6dab7a0ba5" containerName="istio-proxy"
Oct 30 22:31:21.145636 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.145598    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe40461b-fff1-4d46-84d5-262b608028b2" containerName="istio-proxy"
Oct 30 22:31:21.154977 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.152134    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.189522 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.189483    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"]
Oct 30 22:31:21.190089 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.190062    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-8b3e43-openshift-default-dockercfg-t8n4l\""
Oct 30 22:31:21.335248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.327272    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:21.335248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.329634    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istiod-ca-cert\") pod \"e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") " pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.335248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.329712    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-token\") pod \"e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") " pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.335248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.329830    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-workload-certs\") pod \"e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") " pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.335248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.329970    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-workload-socket\") pod \"e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") " pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.335248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.330006    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-envoy\") pod \"e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") " pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.335248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.330054    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cplvh\" (UniqueName: \"kubernetes.io/projected/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-kube-api-access-cplvh\") pod \"e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") " pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.335248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.330110    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-credential-socket\") pod \"e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") " pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.335248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.330412    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-podinfo\") pod \"e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") " pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.335248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.330510    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-data\") pod \"e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") " pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.432243 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.432118    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istiod-ca-cert\") pod \"e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") " pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.432583 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.432533    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-token\") pod \"e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") " pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.432785 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.432767    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-workload-certs\") pod \"e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") " pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.432917 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.432904    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-workload-socket\") pod \"e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") " pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.433023 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.433011    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-envoy\") pod \"e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") " pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.433141 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.433126    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cplvh\" (UniqueName: \"kubernetes.io/projected/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-kube-api-access-cplvh\") pod \"e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") " pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.433308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.433290    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-credential-socket\") pod \"e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") " pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.433638 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.433619    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-podinfo\") pod \"e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") " pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.433847 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.433827    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-data\") pod \"e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") " pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.434600 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.434580    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-data\") pod \"e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") " pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.434731 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.434598    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-workload-socket\") pod \"e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") " pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.435883 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.435211    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istiod-ca-cert\") pod \"e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") " pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.435883 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.435500    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-credential-socket\") pod \"e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") " pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.436311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.436292    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-workload-certs\") pod \"e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") " pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.440299 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.439594    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-envoy\") pod \"e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") " pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.440299 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.440129    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-podinfo\") pod \"e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") " pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.479248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.477302    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cplvh\" (UniqueName: \"kubernetes.io/projected/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-kube-api-access-cplvh\") pod \"e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") " pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.489046 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.489006    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-token\") pod \"e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") " pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.494451 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.494362    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:21.495307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.494882    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:21.872245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.867963    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:21.974624 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.974538    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"]
Oct 30 22:31:21.978370 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:31:21.978340    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a318ba_6fbd_479f_aa9a_e07dc2ab12a8.slice/crio-e05826b9bce92a15a7abb6ecf1220018087733c9b390b8e714c4e96866b355ce WatchSource:0}: Error finding container e05826b9bce92a15a7abb6ecf1220018087733c9b390b8e714c4e96866b355ce: Status 404 returned error can't find the container with id e05826b9bce92a15a7abb6ecf1220018087733c9b390b8e714c4e96866b355ce
Oct 30 22:31:21.981770 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.981732    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:31:21.981897 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.981813    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:31:21.981897 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:21.981853    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:31:22.039138 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:22.038900    4889 patch_prober.go:29] interesting pod/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": read tcp 10.132.0.2:53948->10.132.0.130:15021: read: connection reset by peer" start-of-body=
Oct 30 22:31:22.039364 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:22.039184    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq" podUID="c81505a9-d460-44ad-ac11-0195c6f8bce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": read tcp 10.132.0.2:53948->10.132.0.130:15021: read: connection reset by peer"
Oct 30 22:31:22.995496 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:22.995458    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p" event={"ID":"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8","Type":"ContainerStarted","Data":"d73d742efb808e8ea43f9a3b9ae9c0e1a94061777bdd868b121bc998e8c61570"}
Oct 30 22:31:22.995983 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:22.995516    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p" event={"ID":"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8","Type":"ContainerStarted","Data":"e05826b9bce92a15a7abb6ecf1220018087733c9b390b8e714c4e96866b355ce"}
Oct 30 22:31:22.997337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:22.997311    4889 generic.go:356] "Generic (PLEG): container finished" podID="c81505a9-d460-44ad-ac11-0195c6f8bce5" containerID="e8ec72c89a7d46dcb289290ddf46b65a1985f6f3d594536b2425667a213a49bd" exitCode=0
Oct 30 22:31:22.997438 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:22.997383    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq" event={"ID":"c81505a9-d460-44ad-ac11-0195c6f8bce5","Type":"ContainerDied","Data":"e8ec72c89a7d46dcb289290ddf46b65a1985f6f3d594536b2425667a213a49bd"}
Oct 30 22:31:22.997749 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:22.997735    4889 scope.go:122] "RemoveContainer" containerID="e8ec72c89a7d46dcb289290ddf46b65a1985f6f3d594536b2425667a213a49bd"
Oct 30 22:31:22.999446 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:22.999418    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:31:22.999564 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:22.999499    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:31:22.999564 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:22.999546    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:31:23.057170 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:23.057121    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p" podStartSLOduration=2.05710744 podStartE2EDuration="2.05710744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:23.053269498 +0000 UTC m=+1468.409635806" watchObservedRunningTime="2026-10-30 22:31:23.05710744 +0000 UTC m=+1468.413473754"
Oct 30 22:31:23.496195 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:23.496163    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:23.502758 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:23.502735    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:23.620251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:23.609879    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:24.008084 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:24.008051    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq" event={"ID":"c81505a9-d460-44ad-ac11-0195c6f8bce5","Type":"ContainerStarted","Data":"c94038f6b9536dfb03ef18c1470371191614a9103ee28f693a26190ec427ddfd"}
Oct 30 22:31:24.008612 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:24.008497    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:24.032249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:24.028660    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:31:24.429303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:24.428620    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:31:24.429303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:24.428743    4889 patch_prober.go:29] interesting pod/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:31:24.429303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:24.428785    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq" podUID="c81505a9-d460-44ad-ac11-0195c6f8bce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 22:31:25.435271 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:25.435183    4889 patch_prober.go:29] interesting pod/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:31:25.435767 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:25.435299    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq" podUID="c81505a9-d460-44ad-ac11-0195c6f8bce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 22:31:25.592494 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:25.592456    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:25.729297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:25.719716    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"]
Oct 30 22:31:25.932096 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:25.932058    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:25.988717 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:25.988641    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"]
Oct 30 22:31:26.078266 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.068387    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35eb0630-0a80-41fc-ab11-6d2577213921-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.078266 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.077779    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.078266 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.077844    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.078827 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.078806    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.080034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.079990    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35eb0630-0a80-41fc-ab11-6d2577213921-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.080884 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.080851    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhxbq\" (UniqueName: \"kubernetes.io/projected/35eb0630-0a80-41fc-ab11-6d2577213921-kube-api-access-mhxbq\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.081121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.081103    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35eb0630-0a80-41fc-ab11-6d2577213921-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.081441 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.081407    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.081612 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.081583    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.081923 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.081891    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.187743 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.187708    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhxbq\" (UniqueName: \"kubernetes.io/projected/35eb0630-0a80-41fc-ab11-6d2577213921-kube-api-access-mhxbq\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.187743 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.187741    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35eb0630-0a80-41fc-ab11-6d2577213921-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.187990 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.187853    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.187990 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.187883    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.188109 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.188083    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.188273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.188211    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35eb0630-0a80-41fc-ab11-6d2577213921-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.188375 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.188285    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.188375 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.188287    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.188375 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.188342    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.188555 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.188487    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.188610 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.188580    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35eb0630-0a80-41fc-ab11-6d2577213921-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.188692 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.188671    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35eb0630-0a80-41fc-ab11-6d2577213921-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.188751 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.188687    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.188813 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.188789    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.191093 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.191070    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35eb0630-0a80-41fc-ab11-6d2577213921-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.191191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.191072    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.240292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.235369    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35eb0630-0a80-41fc-ab11-6d2577213921-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.240292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.238954    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhxbq\" (UniqueName: \"kubernetes.io/projected/35eb0630-0a80-41fc-ab11-6d2577213921-kube-api-access-mhxbq\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.406846 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.406816    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:26.434803 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.434769    4889 patch_prober.go:29] interesting pod/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:31:26.434953 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.434826    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq" podUID="c81505a9-d460-44ad-ac11-0195c6f8bce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 22:31:26.758343 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:31:26.758313    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35eb0630_0a80_41fc_ab11_6d2577213921.slice/crio-f3dfa278436786e524557cd3fa05e99d7d6eaaec3c794c93de350468c664ee1b WatchSource:0}: Error finding container f3dfa278436786e524557cd3fa05e99d7d6eaaec3c794c93de350468c664ee1b: Status 404 returned error can't find the container with id f3dfa278436786e524557cd3fa05e99d7d6eaaec3c794c93de350468c664ee1b
Oct 30 22:31:26.758737 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.758477    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"]
Oct 30 22:31:26.761925 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.761892    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:31:26.762038 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.761974    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:31:26.762038 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:26.762008    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:31:27.098111 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:27.098032    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp" event={"ID":"35eb0630-0a80-41fc-ab11-6d2577213921","Type":"ContainerStarted","Data":"7a75e86e8983ed0303c752dc60bd99135a363b39c1bdccb7b47715d8702d5710"}
Oct 30 22:31:27.098111 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:27.098080    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp" event={"ID":"35eb0630-0a80-41fc-ab11-6d2577213921","Type":"ContainerStarted","Data":"f3dfa278436786e524557cd3fa05e99d7d6eaaec3c794c93de350468c664ee1b"}
Oct 30 22:31:27.407038 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:27.406961    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:27.408271 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:27.408195    4889 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 22:31:27.408271 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:27.408256    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp" podUID="35eb0630-0a80-41fc-ab11-6d2577213921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 22:31:27.430246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:27.428576    4889 patch_prober.go:29] interesting pod/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:31:27.430246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:27.428632    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq" podUID="c81505a9-d460-44ad-ac11-0195c6f8bce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 22:31:27.769811 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:27.769769    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:27.859738 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:27.859675    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp" podStartSLOduration=2.859656266 podStartE2EDuration="2.859656266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:27.166793573 +0000 UTC m=+1472.523159885" watchObservedRunningTime="2026-10-30 22:31:27.859656266 +0000 UTC m=+1473.216022580"
Oct 30 22:31:27.864124 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:27.864099    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv"]
Oct 30 22:31:27.868688 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:27.868672    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv"
Oct 30 22:31:27.886483 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:27.886454    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8b3e43\""
Oct 30 22:31:27.889474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:27.889449    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv"]
Oct 30 22:31:27.889605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:27.889593    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8b3e43-dockercfg-wxzjw\""
Oct 30 22:31:28.126337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:28.118326    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmnbl\" (UniqueName: \"kubernetes.io/projected/31998a4b-a2ae-4128-8a93-916f9247f06d-kube-api-access-qmnbl\") pod \"payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv\" (UID: \"31998a4b-a2ae-4128-8a93-916f9247f06d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv"
Oct 30 22:31:28.126337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:28.118486    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31998a4b-a2ae-4128-8a93-916f9247f06d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv\" (UID: \"31998a4b-a2ae-4128-8a93-916f9247f06d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv"
Oct 30 22:31:28.221277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:28.167577    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:28.316247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:28.311590    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmnbl\" (UniqueName: \"kubernetes.io/projected/31998a4b-a2ae-4128-8a93-916f9247f06d-kube-api-access-qmnbl\") pod \"payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv\" (UID: \"31998a4b-a2ae-4128-8a93-916f9247f06d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv"
Oct 30 22:31:28.316247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:28.311881    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31998a4b-a2ae-4128-8a93-916f9247f06d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv\" (UID: \"31998a4b-a2ae-4128-8a93-916f9247f06d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv"
Oct 30 22:31:28.316247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:28.313776    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31998a4b-a2ae-4128-8a93-916f9247f06d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv\" (UID: \"31998a4b-a2ae-4128-8a93-916f9247f06d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv"
Oct 30 22:31:28.358309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:28.333852    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x"]
Oct 30 22:31:28.358309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:28.342976    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x"
Oct 30 22:31:28.373534 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:28.359534    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmnbl\" (UniqueName: \"kubernetes.io/projected/31998a4b-a2ae-4128-8a93-916f9247f06d-kube-api-access-qmnbl\") pod \"payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv\" (UID: \"31998a4b-a2ae-4128-8a93-916f9247f06d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv"
Oct 30 22:31:28.373534 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:28.373322    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x"]
Oct 30 22:31:28.408742 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:28.408702    4889 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 22:31:28.408949 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:28.408789    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp" podUID="35eb0630-0a80-41fc-ab11-6d2577213921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 22:31:28.417284 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:28.414844    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/337ddd2a-f009-47bd-a214-9ca32e15c9f9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x\" (UID: \"337ddd2a-f009-47bd-a214-9ca32e15c9f9\") " pod="openshift-ingress/payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x"
Oct 30 22:31:28.417284 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:28.415090    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwkth\" (UniqueName: \"kubernetes.io/projected/337ddd2a-f009-47bd-a214-9ca32e15c9f9-kube-api-access-qwkth\") pod \"payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x\" (UID: \"337ddd2a-f009-47bd-a214-9ca32e15c9f9\") " pod="openshift-ingress/payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x"
Oct 30 22:31:28.434251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:28.431468    4889 patch_prober.go:29] interesting pod/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:31:28.434251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:28.431526    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq" podUID="c81505a9-d460-44ad-ac11-0195c6f8bce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 22:31:28.487083 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:28.487054    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv"
Oct 30 22:31:28.533252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:28.518379    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/337ddd2a-f009-47bd-a214-9ca32e15c9f9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x\" (UID: \"337ddd2a-f009-47bd-a214-9ca32e15c9f9\") " pod="openshift-ingress/payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x"
Oct 30 22:31:28.533252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:28.518587    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwkth\" (UniqueName: \"kubernetes.io/projected/337ddd2a-f009-47bd-a214-9ca32e15c9f9-kube-api-access-qwkth\") pod \"payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x\" (UID: \"337ddd2a-f009-47bd-a214-9ca32e15c9f9\") " pod="openshift-ingress/payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x"
Oct 30 22:31:28.533252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:28.520272    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/337ddd2a-f009-47bd-a214-9ca32e15c9f9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x\" (UID: \"337ddd2a-f009-47bd-a214-9ca32e15c9f9\") " pod="openshift-ingress/payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x"
Oct 30 22:31:28.571960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:28.571925    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwkth\" (UniqueName: \"kubernetes.io/projected/337ddd2a-f009-47bd-a214-9ca32e15c9f9-kube-api-access-qwkth\") pod \"payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x\" (UID: \"337ddd2a-f009-47bd-a214-9ca32e15c9f9\") " pod="openshift-ingress/payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x"
Oct 30 22:31:28.701308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:28.701279    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x"
Oct 30 22:31:28.868595 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:28.868563    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv"]
Oct 30 22:31:28.870155 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:31:28.870109    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31998a4b_a2ae_4128_8a93_916f9247f06d.slice/crio-aa1974a6282567762edaed3d680c3def5cdedf50dcf008978695d7b6a3fbb320 WatchSource:0}: Error finding container aa1974a6282567762edaed3d680c3def5cdedf50dcf008978695d7b6a3fbb320: Status 404 returned error can't find the container with id aa1974a6282567762edaed3d680c3def5cdedf50dcf008978695d7b6a3fbb320
Oct 30 22:31:29.166525 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:31:29.166483    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod337ddd2a_f009_47bd_a214_9ca32e15c9f9.slice/crio-558778189b23e23340aff38789d1d5c2a6336d4206dba3928447ca20d4444fa2 WatchSource:0}: Error finding container 558778189b23e23340aff38789d1d5c2a6336d4206dba3928447ca20d4444fa2: Status 404 returned error can't find the container with id 558778189b23e23340aff38789d1d5c2a6336d4206dba3928447ca20d4444fa2
Oct 30 22:31:29.169506 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:29.169479    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x"]
Oct 30 22:31:29.382304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:29.382176    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x" event={"ID":"337ddd2a-f009-47bd-a214-9ca32e15c9f9","Type":"ContainerStarted","Data":"dd0718a5be3afc2b162a03fa0e252a5f7f7a2512a6dc8ffe229591d42eb3e2d6"}
Oct 30 22:31:29.382304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:29.382279    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x" event={"ID":"337ddd2a-f009-47bd-a214-9ca32e15c9f9","Type":"ContainerStarted","Data":"558778189b23e23340aff38789d1d5c2a6336d4206dba3928447ca20d4444fa2"}
Oct 30 22:31:29.382800 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:29.382778    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x"
Oct 30 22:31:29.385833 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:29.385804    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv" event={"ID":"31998a4b-a2ae-4128-8a93-916f9247f06d","Type":"ContainerStarted","Data":"8b7be5263ac5151279d5331856f84001d8244240d82f53f99ba5dc9f1e6b03aa"}
Oct 30 22:31:29.385945 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:29.385854    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv" event={"ID":"31998a4b-a2ae-4128-8a93-916f9247f06d","Type":"ContainerStarted","Data":"aa1974a6282567762edaed3d680c3def5cdedf50dcf008978695d7b6a3fbb320"}
Oct 30 22:31:29.408033 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:29.408009    4889 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 22:31:29.408142 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:29.408059    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp" podUID="35eb0630-0a80-41fc-ab11-6d2577213921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 22:31:29.434138 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:29.428334    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x" podStartSLOduration=2.428315265 podStartE2EDuration="2.428315265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:29.426738213 +0000 UTC m=+1474.783104529" watchObservedRunningTime="2026-10-30 22:31:29.428315265 +0000 UTC m=+1474.784681584"
Oct 30 22:31:29.448250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:29.435015    4889 patch_prober.go:29] interesting pod/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:31:29.448250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:29.435073    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq" podUID="c81505a9-d460-44ad-ac11-0195c6f8bce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 22:31:29.487985 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:29.487685    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv" podStartSLOduration=2.487667705 podStartE2EDuration="2.487667705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:29.485604612 +0000 UTC m=+1474.841970927" watchObservedRunningTime="2026-10-30 22:31:29.487667705 +0000 UTC m=+1474.844034017"
Oct 30 22:31:30.133250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:30.132747    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:30.361253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:30.360160    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:30.412310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:30.409681    4889 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 22:31:30.412310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:30.409739    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp" podUID="35eb0630-0a80-41fc-ab11-6d2577213921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 22:31:30.445342 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:30.440257    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv"
Oct 30 22:31:30.455254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:30.454409    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:31:30.456296 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:30.455911    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:31:30.457147 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:30.457127    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:31:31.409690 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:31.409649    4889 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 22:31:31.410277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:31.410242    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp" podUID="35eb0630-0a80-41fc-ab11-6d2577213921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 22:31:32.407471 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:32.407437    4889 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 22:31:32.407657 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:32.407493    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp" podUID="35eb0630-0a80-41fc-ab11-6d2577213921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 22:31:32.457861 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:32.457828    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:33.408090 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:33.408055    4889 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 22:31:33.408307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:33.408107    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp" podUID="35eb0630-0a80-41fc-ab11-6d2577213921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 22:31:33.986932 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:33.986864    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9"]
Oct 30 22:31:34.108941 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:34.108907    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:34.544632 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:34.544587    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9" podUID="3a7d79a6-3485-49c6-98f9-e83552ec94e7" containerName="maas-api" containerID="cri-o://377d6daef8f8d3a183db1d630419ff74c53e4ba180bc0833a469ec75d02e982c" gracePeriod=30
Oct 30 22:31:34.553264 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:34.553213    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:34.559636 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:34.559610    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:34.598857 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:34.598271    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:31:34.975558 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:34.975528    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:34.999093 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:34.999064    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9"
Oct 30 22:31:35.016424 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.016397    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a7d79a6-3485-49c6-98f9-e83552ec94e7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7d79a6-3485-49c6-98f9-e83552ec94e7-openshift-service-ca\") pod \"3a7d79a6-3485-49c6-98f9-e83552ec94e7\" (UID: \"3a7d79a6-3485-49c6-98f9-e83552ec94e7\") "
Oct 30 22:31:35.016581 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.016442    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a7d79a6-3485-49c6-98f9-e83552ec94e7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a7d79a6-3485-49c6-98f9-e83552ec94e7-metrics-certs\") pod \"3a7d79a6-3485-49c6-98f9-e83552ec94e7\" (UID: \"3a7d79a6-3485-49c6-98f9-e83552ec94e7\") "
Oct 30 22:31:35.016581 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.016482    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a7d79a6-3485-49c6-98f9-e83552ec94e7-kube-api-access-jfldk\" (UniqueName: \"kubernetes.io/projected/3a7d79a6-3485-49c6-98f9-e83552ec94e7-kube-api-access-jfldk\") pod \"3a7d79a6-3485-49c6-98f9-e83552ec94e7\" (UID: \"3a7d79a6-3485-49c6-98f9-e83552ec94e7\") "
Oct 30 22:31:35.016581 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.016517    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a7d79a6-3485-49c6-98f9-e83552ec94e7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a7d79a6-3485-49c6-98f9-e83552ec94e7-maas-api-tls\") pod \"3a7d79a6-3485-49c6-98f9-e83552ec94e7\" (UID: \"3a7d79a6-3485-49c6-98f9-e83552ec94e7\") "
Oct 30 22:31:35.016832 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.016804    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a7d79a6-3485-49c6-98f9-e83552ec94e7-openshift-service-ca" pod "3a7d79a6-3485-49c6-98f9-e83552ec94e7" (UID: "3a7d79a6-3485-49c6-98f9-e83552ec94e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:35.017307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.017282    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7d79a6-3485-49c6-98f9-e83552ec94e7-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:35.018991 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.018957    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a7d79a6-3485-49c6-98f9-e83552ec94e7-metrics-certs" pod "3a7d79a6-3485-49c6-98f9-e83552ec94e7" (UID: "3a7d79a6-3485-49c6-98f9-e83552ec94e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:35.019075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.018983    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a7d79a6-3485-49c6-98f9-e83552ec94e7-maas-api-tls" pod "3a7d79a6-3485-49c6-98f9-e83552ec94e7" (UID: "3a7d79a6-3485-49c6-98f9-e83552ec94e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:35.019075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.019015    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a7d79a6-3485-49c6-98f9-e83552ec94e7-kube-api-access-jfldk" pod "3a7d79a6-3485-49c6-98f9-e83552ec94e7" (UID: "3a7d79a6-3485-49c6-98f9-e83552ec94e7"). InnerVolumeSpecName "kube-api-access-jfldk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:35.075071 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.074986    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m"]
Oct 30 22:31:35.075983 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.075958    4889 cpu_manager.go:414] "containerMap: removing container" podUID="3a7d79a6-3485-49c6-98f9-e83552ec94e7" containerName="maas-api"
Oct 30 22:31:35.075983 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.075983    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a7d79a6-3485-49c6-98f9-e83552ec94e7" containerName="maas-api"
Oct 30 22:31:35.076281 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.076247    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a7d79a6-3485-49c6-98f9-e83552ec94e7" containerName="maas-api"
Oct 30 22:31:35.080515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.080489    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m"
Oct 30 22:31:35.100925 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.097458    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-d9f1c2-serving-cert\""
Oct 30 22:31:35.105178 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.105151    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m"]
Oct 30 22:31:35.118784 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.118755    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh7qj\" (UniqueName: \"kubernetes.io/projected/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-kube-api-access-dh7qj\") pod \"maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m\" (UID: \"0589bfca-27f5-4ba4-93af-9a0ffd07ffb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m"
Oct 30 22:31:35.119016 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.118993    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m\" (UID: \"0589bfca-27f5-4ba4-93af-9a0ffd07ffb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m"
Oct 30 22:31:35.119278 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.119257    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m\" (UID: \"0589bfca-27f5-4ba4-93af-9a0ffd07ffb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m"
Oct 30 22:31:35.119592 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.119566    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m\" (UID: \"0589bfca-27f5-4ba4-93af-9a0ffd07ffb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m"
Oct 30 22:31:35.121339 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.119733    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a7d79a6-3485-49c6-98f9-e83552ec94e7-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:35.121339 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.119755    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfldk\" (UniqueName: \"kubernetes.io/projected/3a7d79a6-3485-49c6-98f9-e83552ec94e7-kube-api-access-jfldk\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:35.121339 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.119770    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a7d79a6-3485-49c6-98f9-e83552ec94e7-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:31:35.159031 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.159000    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:35.221125 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.221091    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m\" (UID: \"0589bfca-27f5-4ba4-93af-9a0ffd07ffb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m"
Oct 30 22:31:35.221379 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.221359    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh7qj\" (UniqueName: \"kubernetes.io/projected/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-kube-api-access-dh7qj\") pod \"maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m\" (UID: \"0589bfca-27f5-4ba4-93af-9a0ffd07ffb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m"
Oct 30 22:31:35.221454 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.221407    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m\" (UID: \"0589bfca-27f5-4ba4-93af-9a0ffd07ffb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m"
Oct 30 22:31:35.221529 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.221514    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m\" (UID: \"0589bfca-27f5-4ba4-93af-9a0ffd07ffb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m"
Oct 30 22:31:35.222146 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.222111    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m\" (UID: \"0589bfca-27f5-4ba4-93af-9a0ffd07ffb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m"
Oct 30 22:31:35.223675 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.223656    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m\" (UID: \"0589bfca-27f5-4ba4-93af-9a0ffd07ffb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m"
Oct 30 22:31:35.223934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.223918    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m\" (UID: \"0589bfca-27f5-4ba4-93af-9a0ffd07ffb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m"
Oct 30 22:31:35.240460 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.240433    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh7qj\" (UniqueName: \"kubernetes.io/projected/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-kube-api-access-dh7qj\") pod \"maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m\" (UID: \"0589bfca-27f5-4ba4-93af-9a0ffd07ffb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m"
Oct 30 22:31:35.371304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.367767    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:35.412961 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.412927    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m"
Oct 30 22:31:35.560165 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.560130    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:35.562458 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.562432    4889 generic.go:356] "Generic (PLEG): container finished" podID="3a7d79a6-3485-49c6-98f9-e83552ec94e7" containerID="377d6daef8f8d3a183db1d630419ff74c53e4ba180bc0833a469ec75d02e982c" exitCode=0
Oct 30 22:31:35.562604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.562479    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9"
Oct 30 22:31:35.562604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.562546    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9" event={"ID":"3a7d79a6-3485-49c6-98f9-e83552ec94e7","Type":"ContainerDied","Data":"377d6daef8f8d3a183db1d630419ff74c53e4ba180bc0833a469ec75d02e982c"}
Oct 30 22:31:35.562604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.562582    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9" event={"ID":"3a7d79a6-3485-49c6-98f9-e83552ec94e7","Type":"ContainerDied","Data":"944417d64afc46c750d26701b22390066438a828bb0f1f0f1ed5eea86583f307"}
Oct 30 22:31:35.562604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.562603    4889 scope.go:122] "RemoveContainer" containerID="377d6daef8f8d3a183db1d630419ff74c53e4ba180bc0833a469ec75d02e982c"
Oct 30 22:31:35.583910 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.583881    4889 scope.go:122] "RemoveContainer" containerID="377d6daef8f8d3a183db1d630419ff74c53e4ba180bc0833a469ec75d02e982c"
Oct 30 22:31:35.584710 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:31:35.584669    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"377d6daef8f8d3a183db1d630419ff74c53e4ba180bc0833a469ec75d02e982c\": container with ID starting with 377d6daef8f8d3a183db1d630419ff74c53e4ba180bc0833a469ec75d02e982c not found: ID does not exist" containerID="377d6daef8f8d3a183db1d630419ff74c53e4ba180bc0833a469ec75d02e982c"
Oct 30 22:31:35.584800 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.584722    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377d6daef8f8d3a183db1d630419ff74c53e4ba180bc0833a469ec75d02e982c"} err="failed to get container status \"377d6daef8f8d3a183db1d630419ff74c53e4ba180bc0833a469ec75d02e982c\": rpc error: code = NotFound desc = could not find container \"377d6daef8f8d3a183db1d630419ff74c53e4ba180bc0833a469ec75d02e982c\": container with ID starting with 377d6daef8f8d3a183db1d630419ff74c53e4ba180bc0833a469ec75d02e982c not found: ID does not exist"
Oct 30 22:31:35.649058 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.648993    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9"]
Oct 30 22:31:35.658175 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.657995    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-f5dz9"]
Oct 30 22:31:35.729946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.729911    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:35.753157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:35.753120    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m"]
Oct 30 22:31:35.753760 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:31:35.753731    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0589bfca_27f5_4ba4_93af_9a0ffd07ffb6.slice/crio-2f1096eb0acf0b752a8cc15866a30f721f0c689db6aa2fb24aaa5da81526f1c6 WatchSource:0}: Error finding container 2f1096eb0acf0b752a8cc15866a30f721f0c689db6aa2fb24aaa5da81526f1c6: Status 404 returned error can't find the container with id 2f1096eb0acf0b752a8cc15866a30f721f0c689db6aa2fb24aaa5da81526f1c6
Oct 30 22:31:36.573778 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:36.573743    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m" event={"ID":"0589bfca-27f5-4ba4-93af-9a0ffd07ffb6","Type":"ContainerStarted","Data":"340dcafb13d15c16c838d1f443419dc7b2beda283c0d992b4078cb67412cd1a0"}
Oct 30 22:31:36.574297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:36.573798    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m" event={"ID":"0589bfca-27f5-4ba4-93af-9a0ffd07ffb6","Type":"ContainerStarted","Data":"2f1096eb0acf0b752a8cc15866a30f721f0c689db6aa2fb24aaa5da81526f1c6"}
Oct 30 22:31:36.574297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:36.573843    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m"
Oct 30 22:31:36.614432 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:36.614384    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m" podStartSLOduration=1.253450681 podStartE2EDuration="1.614370721s" totalImagesPullingTime="360.92004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:36.6114349 +0000 UTC m=+1481.967801215" watchObservedRunningTime="2026-10-30 22:31:36.614370721 +0000 UTC m=+1481.970737034"
Oct 30 22:31:36.854674 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:36.854577    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a7d79a6-3485-49c6-98f9-e83552ec94e7" path="/var/lib/kubelet/pods/3a7d79a6-3485-49c6-98f9-e83552ec94e7/volumes"
Oct 30 22:31:37.363250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:37.357920    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:39.511270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:39.511148    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:39.745250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:39.743968    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:40.234245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:40.233023    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:40.303122 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:31:40.303082    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:31:40.454250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:40.446610    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x"
Oct 30 22:31:40.732773 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:40.732737    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:41.324204 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:41.324158    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:41.482080 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:41.482046    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv"
Oct 30 22:31:42.601751 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:42.601724    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m"
Oct 30 22:31:43.113737 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:43.113701    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:44.914269 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:44.914177    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:46.551252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:46.547534    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:47.785975 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:47.785938    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:48.240170 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:48.240085    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:49.929779 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:49.929738    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:51.599054 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:51.598993    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:53.251537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:53.251493    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:55.194045 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:55.194014    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 22:31:55.196037 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:55.195992    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 22:31:55.224376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:55.224343    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:57.139466 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.139432    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"]
Oct 30 22:31:57.147950 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.147920    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.163956 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.163919    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4f7439-openshift-default-dockercfg-2rddv\""
Oct 30 22:31:57.169182 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.169148    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"]
Oct 30 22:31:57.326249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.310412    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-data\") pod \"e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") " pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.326249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.310560    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-podinfo\") pod \"e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") " pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.326249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.310616    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzgpx\" (UniqueName: \"kubernetes.io/projected/41d10d03-c9af-4c80-b5c5-692d1a3953af-kube-api-access-nzgpx\") pod \"e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") " pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.326249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.310677    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-token\") pod \"e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") " pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.326249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.310747    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-credential-socket\") pod \"e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") " pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.326249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.310786    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-workload-socket\") pod \"e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") " pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.326249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.310895    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-workload-certs\") pod \"e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") " pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.326249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.310919    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-envoy\") pod \"e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") " pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.326249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.310946    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d10d03-c9af-4c80-b5c5-692d1a3953af-istiod-ca-cert\") pod \"e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") " pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.432353 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.429715    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:57.432353 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.430985    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-workload-certs\") pod \"e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") " pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.432353 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.431036    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-envoy\") pod \"e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") " pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.432353 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.431081    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d10d03-c9af-4c80-b5c5-692d1a3953af-istiod-ca-cert\") pod \"e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") " pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.432353 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.431323    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-data\") pod \"e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") " pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.432353 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.431457    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-podinfo\") pod \"e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") " pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.432353 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.431514    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzgpx\" (UniqueName: \"kubernetes.io/projected/41d10d03-c9af-4c80-b5c5-692d1a3953af-kube-api-access-nzgpx\") pod \"e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") " pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.432353 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.431577    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-token\") pod \"e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") " pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.432353 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.431659    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-credential-socket\") pod \"e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") " pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.432353 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.431703    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-workload-socket\") pod \"e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") " pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.433097 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.432546    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-data\") pod \"e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") " pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.433097 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.432818    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-workload-certs\") pod \"e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") " pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.441368 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.436300    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-workload-socket\") pod \"e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") " pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.454332 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.446666    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-credential-socket\") pod \"e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") " pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.454332 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.447337    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d10d03-c9af-4c80-b5c5-692d1a3953af-istiod-ca-cert\") pod \"e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") " pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.466297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.458598    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-envoy\") pod \"e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") " pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.561245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.558063    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-podinfo\") pod \"e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") " pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.608400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.590820    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-token\") pod \"e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") " pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.608400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.591370    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzgpx\" (UniqueName: \"kubernetes.io/projected/41d10d03-c9af-4c80-b5c5-692d1a3953af-kube-api-access-nzgpx\") pod \"e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") " pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:57.778053 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:57.777724    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:58.113872 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:31:58.113840    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d10d03_c9af_4c80_b5c5_692d1a3953af.slice/crio-ccd21f15d77f00099b71bce368f7a4ef9434dcda95fc4b9ee6414a5ee2a66e3b WatchSource:0}: Error finding container ccd21f15d77f00099b71bce368f7a4ef9434dcda95fc4b9ee6414a5ee2a66e3b: Status 404 returned error can't find the container with id ccd21f15d77f00099b71bce368f7a4ef9434dcda95fc4b9ee6414a5ee2a66e3b
Oct 30 22:31:58.114015 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:58.113919    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"]
Oct 30 22:31:58.117002 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:58.116968    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:31:58.117102 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:58.117039    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:31:58.117102 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:58.117069    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:31:58.190189 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:58.190153    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj" event={"ID":"41d10d03-c9af-4c80-b5c5-692d1a3953af","Type":"ContainerStarted","Data":"ccd21f15d77f00099b71bce368f7a4ef9434dcda95fc4b9ee6414a5ee2a66e3b"}
Oct 30 22:31:59.199981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:59.199948    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj" event={"ID":"41d10d03-c9af-4c80-b5c5-692d1a3953af","Type":"ContainerStarted","Data":"4135272e3e3db1908e8789c1be14aff438670105bfaebf377c19742d1df753d3"}
Oct 30 22:31:59.251697 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:59.251633    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj" podStartSLOduration=2.251615475 podStartE2EDuration="2.251615475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:59.249964312 +0000 UTC m=+1504.606330627" watchObservedRunningTime="2026-10-30 22:31:59.251615475 +0000 UTC m=+1504.607981790"
Oct 30 22:31:59.437642 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:59.437543    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:59.778576 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:59.778544    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:31:59.785414 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:31:59.785385    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:32:00.209572 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:00.209490    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:32:00.210550 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:00.210533    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:32:01.391307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:01.382081    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:01.843217 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:01.843176    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:02.050407 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:02.050359    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:03.557980 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:03.557943    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:04.470504 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:04.470407    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:04.671683 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:04.671652    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:05.760249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:05.758733    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:07.617097 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:07.617061    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:08.539477 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:08.539312    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:09.538152 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:09.538113    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:10.305469 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:32:10.305433    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:32:11.496330 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:11.496295    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:11.697015 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:11.696978    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:11.982778 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:11.982671    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:13.485501 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:13.478867    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:15.244720 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:15.244680    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:16.172248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:16.162753    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:16.999040 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:16.998997    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:17.166137 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:17.166100    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:19.093484 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:19.093448    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:20.472408 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:20.472376    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq"]
Oct 30 22:32:20.511374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:20.511342    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx"]
Oct 30 22:32:20.681816 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:20.681772    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:20.846845 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:20.846812    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:21.028070 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:21.028026    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:23.029670 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:23.029628    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:23.215015 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:23.214969    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:23.313308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:23.313201    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"]
Oct 30 22:32:23.314237 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:23.314174    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn" podUID="62c75d3c-bba8-4a52-a0cf-f76a2b4b8495" containerName="istio-proxy" containerID="cri-o://73a127efa766b7a42a4d634d676dc9b91d384681d01ab479f1af4721ac308b13" gracePeriod=30
Oct 30 22:32:23.493258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:23.492645    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:23.780286 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:23.780209    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:24.336251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:24.313689    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:24.790960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:24.790160    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:25.472795 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:25.472750    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq" podUID="6a26178c-e117-4fcf-be0d-7a8fcf1ddef3" containerName="payload-processing" containerID="cri-o://0ea3df77d9f37b970123dc7112573ce5cda03646d8293fecab761309b221cc9f" gracePeriod=25
Oct 30 22:32:25.512039 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:25.512006    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx" podUID="ea29a485-bd51-436a-a6a4-03fc95118b57" containerName="payload-pre-processing" containerID="cri-o://0cb162f0eb6aa5ebb9faeadc388b86b4190fd32e5148dba14fed18a05412e637" gracePeriod=25
Oct 30 22:32:25.630382 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:25.630336    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:25.910081 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:25.910053    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx"
Oct 30 22:32:25.957621 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:25.952254    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq"
Oct 30 22:32:26.034488 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.034402    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a26178c-e117-4fcf-be0d-7a8fcf1ddef3-kube-api-access-tb4rw\" (UniqueName: \"kubernetes.io/projected/6a26178c-e117-4fcf-be0d-7a8fcf1ddef3-kube-api-access-tb4rw\") pod \"6a26178c-e117-4fcf-be0d-7a8fcf1ddef3\" (UID: \"6a26178c-e117-4fcf-be0d-7a8fcf1ddef3\") "
Oct 30 22:32:26.034644 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.034571    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea29a485-bd51-436a-a6a4-03fc95118b57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea29a485-bd51-436a-a6a4-03fc95118b57-plugins-config-volume\") pod \"ea29a485-bd51-436a-a6a4-03fc95118b57\" (UID: \"ea29a485-bd51-436a-a6a4-03fc95118b57\") "
Oct 30 22:32:26.034713 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.034698    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea29a485-bd51-436a-a6a4-03fc95118b57-kube-api-access-m86lh\" (UniqueName: \"kubernetes.io/projected/ea29a485-bd51-436a-a6a4-03fc95118b57-kube-api-access-m86lh\") pod \"ea29a485-bd51-436a-a6a4-03fc95118b57\" (UID: \"ea29a485-bd51-436a-a6a4-03fc95118b57\") "
Oct 30 22:32:26.034836 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.034817    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a26178c-e117-4fcf-be0d-7a8fcf1ddef3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a26178c-e117-4fcf-be0d-7a8fcf1ddef3-plugins-config-volume\") pod \"6a26178c-e117-4fcf-be0d-7a8fcf1ddef3\" (UID: \"6a26178c-e117-4fcf-be0d-7a8fcf1ddef3\") "
Oct 30 22:32:26.035093 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.035018    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea29a485-bd51-436a-a6a4-03fc95118b57-plugins-config-volume" pod "ea29a485-bd51-436a-a6a4-03fc95118b57" (UID: "ea29a485-bd51-436a-a6a4-03fc95118b57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:26.035684 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.035658    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea29a485-bd51-436a-a6a4-03fc95118b57-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:32:26.035812 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.035669    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a26178c-e117-4fcf-be0d-7a8fcf1ddef3-plugins-config-volume" pod "6a26178c-e117-4fcf-be0d-7a8fcf1ddef3" (UID: "6a26178c-e117-4fcf-be0d-7a8fcf1ddef3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:26.037406 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.037382    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea29a485-bd51-436a-a6a4-03fc95118b57-kube-api-access-m86lh" pod "ea29a485-bd51-436a-a6a4-03fc95118b57" (UID: "ea29a485-bd51-436a-a6a4-03fc95118b57"). InnerVolumeSpecName "kube-api-access-m86lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:26.037565 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.037539    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a26178c-e117-4fcf-be0d-7a8fcf1ddef3-kube-api-access-tb4rw" pod "6a26178c-e117-4fcf-be0d-7a8fcf1ddef3" (UID: "6a26178c-e117-4fcf-be0d-7a8fcf1ddef3"). InnerVolumeSpecName "kube-api-access-tb4rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:26.137683 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.137650    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a26178c-e117-4fcf-be0d-7a8fcf1ddef3-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:32:26.137683 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.137680    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb4rw\" (UniqueName: \"kubernetes.io/projected/6a26178c-e117-4fcf-be0d-7a8fcf1ddef3-kube-api-access-tb4rw\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:32:26.137683 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.137689    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m86lh\" (UniqueName: \"kubernetes.io/projected/ea29a485-bd51-436a-a6a4-03fc95118b57-kube-api-access-m86lh\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:32:26.581476 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.581436    4889 generic.go:356] "Generic (PLEG): container finished" podID="ea29a485-bd51-436a-a6a4-03fc95118b57" containerID="0cb162f0eb6aa5ebb9faeadc388b86b4190fd32e5148dba14fed18a05412e637" exitCode=0
Oct 30 22:32:26.581934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.581487    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx"
Oct 30 22:32:26.581934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.581520    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx" event={"ID":"ea29a485-bd51-436a-a6a4-03fc95118b57","Type":"ContainerDied","Data":"0cb162f0eb6aa5ebb9faeadc388b86b4190fd32e5148dba14fed18a05412e637"}
Oct 30 22:32:26.581934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.581572    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx" event={"ID":"ea29a485-bd51-436a-a6a4-03fc95118b57","Type":"ContainerDied","Data":"5b72ac43b2ebb1b2e7a576242f4b2b2fd3bb95a17d6b9b1111e457f299f4dfcc"}
Oct 30 22:32:26.581934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.581595    4889 scope.go:122] "RemoveContainer" containerID="0cb162f0eb6aa5ebb9faeadc388b86b4190fd32e5148dba14fed18a05412e637"
Oct 30 22:32:26.583329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.583307    4889 generic.go:356] "Generic (PLEG): container finished" podID="6a26178c-e117-4fcf-be0d-7a8fcf1ddef3" containerID="0ea3df77d9f37b970123dc7112573ce5cda03646d8293fecab761309b221cc9f" exitCode=0
Oct 30 22:32:26.583423 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.583361    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq"
Oct 30 22:32:26.583423 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.583365    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq" event={"ID":"6a26178c-e117-4fcf-be0d-7a8fcf1ddef3","Type":"ContainerDied","Data":"0ea3df77d9f37b970123dc7112573ce5cda03646d8293fecab761309b221cc9f"}
Oct 30 22:32:26.583685 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.583537    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq" event={"ID":"6a26178c-e117-4fcf-be0d-7a8fcf1ddef3","Type":"ContainerDied","Data":"b71f8b221464c427f93e1db436137edb4b55f352787cf755d15a3fb5145f7a9e"}
Oct 30 22:32:26.627436 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.625608    4889 scope.go:122] "RemoveContainer" containerID="0cb162f0eb6aa5ebb9faeadc388b86b4190fd32e5148dba14fed18a05412e637"
Oct 30 22:32:26.627436 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:32:26.626131    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cb162f0eb6aa5ebb9faeadc388b86b4190fd32e5148dba14fed18a05412e637\": container with ID starting with 0cb162f0eb6aa5ebb9faeadc388b86b4190fd32e5148dba14fed18a05412e637 not found: ID does not exist" containerID="0cb162f0eb6aa5ebb9faeadc388b86b4190fd32e5148dba14fed18a05412e637"
Oct 30 22:32:26.627436 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.626216    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cb162f0eb6aa5ebb9faeadc388b86b4190fd32e5148dba14fed18a05412e637"} err="failed to get container status \"0cb162f0eb6aa5ebb9faeadc388b86b4190fd32e5148dba14fed18a05412e637\": rpc error: code = NotFound desc = could not find container \"0cb162f0eb6aa5ebb9faeadc388b86b4190fd32e5148dba14fed18a05412e637\": container with ID starting with 0cb162f0eb6aa5ebb9faeadc388b86b4190fd32e5148dba14fed18a05412e637 not found: ID does not exist"
Oct 30 22:32:26.627436 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.626327    4889 scope.go:122] "RemoveContainer" containerID="0ea3df77d9f37b970123dc7112573ce5cda03646d8293fecab761309b221cc9f"
Oct 30 22:32:26.703472 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.703401    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx"]
Oct 30 22:32:26.772247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.757788    4889 scope.go:122] "RemoveContainer" containerID="0ea3df77d9f37b970123dc7112573ce5cda03646d8293fecab761309b221cc9f"
Oct 30 22:32:26.774509 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:32:26.774455    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ea3df77d9f37b970123dc7112573ce5cda03646d8293fecab761309b221cc9f\": container with ID starting with 0ea3df77d9f37b970123dc7112573ce5cda03646d8293fecab761309b221cc9f not found: ID does not exist" containerID="0ea3df77d9f37b970123dc7112573ce5cda03646d8293fecab761309b221cc9f"
Oct 30 22:32:26.774660 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.774518    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea3df77d9f37b970123dc7112573ce5cda03646d8293fecab761309b221cc9f"} err="failed to get container status \"0ea3df77d9f37b970123dc7112573ce5cda03646d8293fecab761309b221cc9f\": rpc error: code = NotFound desc = could not find container \"0ea3df77d9f37b970123dc7112573ce5cda03646d8293fecab761309b221cc9f\": container with ID starting with 0ea3df77d9f37b970123dc7112573ce5cda03646d8293fecab761309b221cc9f not found: ID does not exist"
Oct 30 22:32:26.885555 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.885504    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-370ec2-7957558899-vkjqx"]
Oct 30 22:32:26.895389 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.895353    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq"]
Oct 30 22:32:26.901318 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:26.901286    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-370ec2-7658c775d7-t69wq"]
Oct 30 22:32:27.918955 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:27.918608    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:28.110541 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.110430    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:28.611911 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.611883    4889 generic.go:356] "Generic (PLEG): container finished" podID="62c75d3c-bba8-4a52-a0cf-f76a2b4b8495" containerID="73a127efa766b7a42a4d634d676dc9b91d384681d01ab479f1af4721ac308b13" exitCode=0
Oct 30 22:32:28.612079 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.611950    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn" event={"ID":"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495","Type":"ContainerDied","Data":"73a127efa766b7a42a4d634d676dc9b91d384681d01ab479f1af4721ac308b13"}
Oct 30 22:32:28.612079 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.612006    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn" event={"ID":"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495","Type":"ContainerDied","Data":"30999e8beeb06668d7083ecf85e3708998c34d8d9a82d01f0f0e626e18224bf0"}
Oct 30 22:32:28.612079 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.612024    4889 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30999e8beeb06668d7083ecf85e3708998c34d8d9a82d01f0f0e626e18224bf0"
Oct 30 22:32:28.612079 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.612037    4889 scope.go:122] "RemoveContainer" containerID="8595aa86978856d85bb6582032e10dcd5561f2ba65c802c5800c469f310a76bd"
Oct 30 22:32:28.669311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.669278    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:32:28.826142 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.826105    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a26178c-e117-4fcf-be0d-7a8fcf1ddef3" path="/var/lib/kubelet/pods/6a26178c-e117-4fcf-be0d-7a8fcf1ddef3/volumes"
Oct 30 22:32:28.827197 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.827171    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea29a485-bd51-436a-a6a4-03fc95118b57" path="/var/lib/kubelet/pods/ea29a485-bd51-436a-a6a4-03fc95118b57/volumes"
Oct 30 22:32:28.835774 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.835746    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istiod-ca-cert\") pod \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") "
Oct 30 22:32:28.836015 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.835995    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-data\") pod \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") "
Oct 30 22:32:28.836165 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.836149    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-podinfo\") pod \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") "
Oct 30 22:32:28.837257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.836670    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istiod-ca-cert" pod "62c75d3c-bba8-4a52-a0cf-f76a2b4b8495" (UID: "62c75d3c-bba8-4a52-a0cf-f76a2b4b8495"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:28.837257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.836836    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-data" pod "62c75d3c-bba8-4a52-a0cf-f76a2b4b8495" (UID: "62c75d3c-bba8-4a52-a0cf-f76a2b4b8495"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:28.837588 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.837567    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-kube-api-access-cxsbm\" (UniqueName: \"kubernetes.io/projected/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-kube-api-access-cxsbm\") pod \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") "
Oct 30 22:32:28.837806 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.837790    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-envoy\") pod \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") "
Oct 30 22:32:28.838007 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.837992    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-credential-socket\") pod \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") "
Oct 30 22:32:28.838455 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.838435    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-workload-certs\") pod \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") "
Oct 30 22:32:28.838618 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.838601    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-token\" (UniqueName: \"kubernetes.io/projected/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-token\") pod \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") "
Oct 30 22:32:28.838751 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.838738    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-workload-socket\") pod \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\" (UID: \"62c75d3c-bba8-4a52-a0cf-f76a2b4b8495\") "
Oct 30 22:32:28.846299 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.839687    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:32:28.846489 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.846469    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:32:28.846832 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.846808    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-workload-socket" pod "62c75d3c-bba8-4a52-a0cf-f76a2b4b8495" (UID: "62c75d3c-bba8-4a52-a0cf-f76a2b4b8495"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:28.847113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.847095    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-workload-certs" pod "62c75d3c-bba8-4a52-a0cf-f76a2b4b8495" (UID: "62c75d3c-bba8-4a52-a0cf-f76a2b4b8495"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:28.870371 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.864410    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-credential-socket" pod "62c75d3c-bba8-4a52-a0cf-f76a2b4b8495" (UID: "62c75d3c-bba8-4a52-a0cf-f76a2b4b8495"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:28.892386 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.886597    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-envoy" pod "62c75d3c-bba8-4a52-a0cf-f76a2b4b8495" (UID: "62c75d3c-bba8-4a52-a0cf-f76a2b4b8495"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:28.898036 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.898003    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-kube-api-access-cxsbm" pod "62c75d3c-bba8-4a52-a0cf-f76a2b4b8495" (UID: "62c75d3c-bba8-4a52-a0cf-f76a2b4b8495"). InnerVolumeSpecName "kube-api-access-cxsbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:28.903409 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.903374    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-token" pod "62c75d3c-bba8-4a52-a0cf-f76a2b4b8495" (UID: "62c75d3c-bba8-4a52-a0cf-f76a2b4b8495"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:28.936258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.928549    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-podinfo" pod "62c75d3c-bba8-4a52-a0cf-f76a2b4b8495" (UID: "62c75d3c-bba8-4a52-a0cf-f76a2b4b8495"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:32:28.964590 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.959535    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxsbm\" (UniqueName: \"kubernetes.io/projected/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-kube-api-access-cxsbm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:32:28.964590 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.959582    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:32:28.964590 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.959599    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:32:28.964590 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.959617    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:32:28.964590 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.959634    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:32:28.964590 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.959650    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:32:28.964590 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:28.959665    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:32:29.096716 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.096615    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:29.265942 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.265905    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:29.444791 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.444755    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"]
Oct 30 22:32:29.445777 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.445748    4889 cpu_manager.go:414] "containerMap: removing container" podUID="ea29a485-bd51-436a-a6a4-03fc95118b57" containerName="payload-pre-processing"
Oct 30 22:32:29.445777 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.445776    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea29a485-bd51-436a-a6a4-03fc95118b57" containerName="payload-pre-processing"
Oct 30 22:32:29.445978 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.445852    4889 cpu_manager.go:414] "containerMap: removing container" podUID="62c75d3c-bba8-4a52-a0cf-f76a2b4b8495" containerName="istio-proxy"
Oct 30 22:32:29.445978 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.445863    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c75d3c-bba8-4a52-a0cf-f76a2b4b8495" containerName="istio-proxy"
Oct 30 22:32:29.445978 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.445878    4889 cpu_manager.go:414] "containerMap: removing container" podUID="62c75d3c-bba8-4a52-a0cf-f76a2b4b8495" containerName="istio-proxy"
Oct 30 22:32:29.445978 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.445886    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c75d3c-bba8-4a52-a0cf-f76a2b4b8495" containerName="istio-proxy"
Oct 30 22:32:29.445978 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.445906    4889 cpu_manager.go:414] "containerMap: removing container" podUID="6a26178c-e117-4fcf-be0d-7a8fcf1ddef3" containerName="payload-processing"
Oct 30 22:32:29.445978 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.445915    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a26178c-e117-4fcf-be0d-7a8fcf1ddef3" containerName="payload-processing"
Oct 30 22:32:29.446284 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.446110    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea29a485-bd51-436a-a6a4-03fc95118b57" containerName="payload-pre-processing"
Oct 30 22:32:29.446284 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.446131    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c75d3c-bba8-4a52-a0cf-f76a2b4b8495" containerName="istio-proxy"
Oct 30 22:32:29.446284 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.446139    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c75d3c-bba8-4a52-a0cf-f76a2b4b8495" containerName="istio-proxy"
Oct 30 22:32:29.446284 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.446147    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a26178c-e117-4fcf-be0d-7a8fcf1ddef3" containerName="payload-processing"
Oct 30 22:32:29.451696 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.451673    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:29.476407 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.476368    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6922d3-openshift-default-dockercfg-5ldsg\""
Oct 30 22:32:29.481945 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.479827    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"]
Oct 30 22:32:29.581485 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.577506    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-podinfo\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:29.581485 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.577625    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-data\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:29.581485 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.577688    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-workload-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:29.581485 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.577763    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-credential-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:29.581485 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.577843    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istiod-ca-cert\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:29.581485 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.577874    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-envoy\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:29.581485 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.577960    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwv79\" (UniqueName: \"kubernetes.io/projected/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-kube-api-access-vwv79\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:29.581485 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.578025    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-workload-certs\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:29.581485 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.578152    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-token\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:29.776310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.722699    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-token\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:29.776310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.722788    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-podinfo\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:29.776310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.722878    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-data\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:29.776310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.722941    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-workload-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:29.776310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.723016    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-credential-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:29.776310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.723094    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istiod-ca-cert\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:29.776310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.723123    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-envoy\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:29.776310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.723206    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwv79\" (UniqueName: \"kubernetes.io/projected/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-kube-api-access-vwv79\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:29.776310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.723292    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-workload-certs\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:29.776310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.723985    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-workload-certs\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:29.776310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.727681    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-credential-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:29.776310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.727943    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-data\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:29.776310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.728178    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-workload-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:29.776310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.734484    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istiod-ca-cert\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:29.896822 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.896686    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-envoy\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:29.955311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.920047    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-token\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:29.955311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.923976    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"
Oct 30 22:32:29.960046 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.960008    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:29.973353 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:29.972069    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-podinfo\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:30.022649 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:30.004817    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwv79\" (UniqueName: \"kubernetes.io/projected/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-kube-api-access-vwv79\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:30.098247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:30.095834    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:30.147881 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:30.147846    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:30.185191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:30.185140    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"]
Oct 30 22:32:30.190248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:30.189958    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2949c143-openshift-default-cc8cc894f-jnvwn"]
Oct 30 22:32:30.465741 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:32:30.465713    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b34996e_28a0_4cbf_b0fa_4615ad9fccaa.slice/crio-9944626a087204c0c1cf4cb5638d6fe97029bbe1fa954d7a7432be80c67ff7d0 WatchSource:0}: Error finding container 9944626a087204c0c1cf4cb5638d6fe97029bbe1fa954d7a7432be80c67ff7d0: Status 404 returned error can't find the container with id 9944626a087204c0c1cf4cb5638d6fe97029bbe1fa954d7a7432be80c67ff7d0
Oct 30 22:32:30.466139 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:30.466117    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"]
Oct 30 22:32:30.468789 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:30.468758    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:32:30.468870 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:30.468822    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:32:30.468870 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:30.468853    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:32:30.549255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:30.543332    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:30.779519 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:30.779486    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c75d3c-bba8-4a52-a0cf-f76a2b4b8495" path="/var/lib/kubelet/pods/62c75d3c-bba8-4a52-a0cf-f76a2b4b8495/volumes"
Oct 30 22:32:30.947278 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:30.947200    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z" event={"ID":"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa","Type":"ContainerStarted","Data":"13a8216f5efc056073c38bf635471fcaae2938ab56d8d1df481b9d5dc2188a75"}
Oct 30 22:32:30.947440 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:30.947307    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z" event={"ID":"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa","Type":"ContainerStarted","Data":"9944626a087204c0c1cf4cb5638d6fe97029bbe1fa954d7a7432be80c67ff7d0"}
Oct 30 22:32:30.997354 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:30.997323    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:31.098493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:31.098405    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:31.103547 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:31.103517    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 22:32:31.103714 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:31.103570    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z" podUID="8b34996e-28a0-4cbf-b0fa-4615ad9fccaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 22:32:31.104603 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:31.104560    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z" podStartSLOduration=2.104535598 podStartE2EDuration="2.104535598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:30.992461512 +0000 UTC m=+1536.348827827" watchObservedRunningTime="2026-10-30 22:32:31.104535598 +0000 UTC m=+1536.460901912"
Oct 30 22:32:31.108903 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:31.108879    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l"]
Oct 30 22:32:31.145050 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:31.141148    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l"
Oct 30 22:32:31.149789 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:31.149191    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l"]
Oct 30 22:32:31.158983 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:31.158955    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-62236a\""
Oct 30 22:32:31.213977 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:31.201919    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-62236a-dockercfg-fg5x4\""
Oct 30 22:32:31.322272 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:31.280702    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:31.379308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:31.327463    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkdx8\" (UniqueName: \"kubernetes.io/projected/e0a34b6d-354a-4b67-8961-c336db52c10e-kube-api-access-bkdx8\") pod \"payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l\" (UID: \"e0a34b6d-354a-4b67-8961-c336db52c10e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l"
Oct 30 22:32:31.379984 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:31.379946    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a34b6d-354a-4b67-8961-c336db52c10e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l\" (UID: \"e0a34b6d-354a-4b67-8961-c336db52c10e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l"
Oct 30 22:32:31.599980 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:31.592796    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a34b6d-354a-4b67-8961-c336db52c10e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l\" (UID: \"e0a34b6d-354a-4b67-8961-c336db52c10e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l"
Oct 30 22:32:31.600531 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:31.600495    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkdx8\" (UniqueName: \"kubernetes.io/projected/e0a34b6d-354a-4b67-8961-c336db52c10e-kube-api-access-bkdx8\") pod \"payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l\" (UID: \"e0a34b6d-354a-4b67-8961-c336db52c10e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l"
Oct 30 22:32:31.609107 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:31.609073    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a34b6d-354a-4b67-8961-c336db52c10e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l\" (UID: \"e0a34b6d-354a-4b67-8961-c336db52c10e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l"
Oct 30 22:32:31.702353 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:31.699006    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf"]
Oct 30 22:32:31.718279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:31.716473    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf"]
Oct 30 22:32:31.718279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:31.716629    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf"
Oct 30 22:32:31.722171 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:31.722139    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkdx8\" (UniqueName: \"kubernetes.io/projected/e0a34b6d-354a-4b67-8961-c336db52c10e-kube-api-access-bkdx8\") pod \"payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l\" (UID: \"e0a34b6d-354a-4b67-8961-c336db52c10e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l"
Oct 30 22:32:31.747250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:31.724750    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:31.807142 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:31.802314    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l"
Oct 30 22:32:31.807142 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:31.806158    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqrkf\" (UniqueName: \"kubernetes.io/projected/8646992a-ed0b-4a46-b781-d9e3f5aff39b-kube-api-access-dqrkf\") pod \"payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf\" (UID: \"8646992a-ed0b-4a46-b781-d9e3f5aff39b\") " pod="openshift-ingress/payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf"
Oct 30 22:32:31.807142 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:31.806378    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8646992a-ed0b-4a46-b781-d9e3f5aff39b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf\" (UID: \"8646992a-ed0b-4a46-b781-d9e3f5aff39b\") " pod="openshift-ingress/payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf"
Oct 30 22:32:31.908127 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:31.908095    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqrkf\" (UniqueName: \"kubernetes.io/projected/8646992a-ed0b-4a46-b781-d9e3f5aff39b-kube-api-access-dqrkf\") pod \"payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf\" (UID: \"8646992a-ed0b-4a46-b781-d9e3f5aff39b\") " pod="openshift-ingress/payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf"
Oct 30 22:32:31.908299 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:31.908214    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8646992a-ed0b-4a46-b781-d9e3f5aff39b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf\" (UID: \"8646992a-ed0b-4a46-b781-d9e3f5aff39b\") " pod="openshift-ingress/payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf"
Oct 30 22:32:31.908947 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:31.908899    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8646992a-ed0b-4a46-b781-d9e3f5aff39b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf\" (UID: \"8646992a-ed0b-4a46-b781-d9e3f5aff39b\") " pod="openshift-ingress/payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf"
Oct 30 22:32:31.941962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:31.941923    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqrkf\" (UniqueName: \"kubernetes.io/projected/8646992a-ed0b-4a46-b781-d9e3f5aff39b-kube-api-access-dqrkf\") pod \"payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf\" (UID: \"8646992a-ed0b-4a46-b781-d9e3f5aff39b\") " pod="openshift-ingress/payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf"
Oct 30 22:32:32.103466 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:32.103428    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:32.145391 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:32.145360    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l"]
Oct 30 22:32:32.147090 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:32:32.147059    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a34b6d_354a_4b67_8961_c336db52c10e.slice/crio-af54f9a911b73d52c5ad379ba049ed6c5098c9cde37d6516ee29894c2845a0a3 WatchSource:0}: Error finding container af54f9a911b73d52c5ad379ba049ed6c5098c9cde37d6516ee29894c2845a0a3: Status 404 returned error can't find the container with id af54f9a911b73d52c5ad379ba049ed6c5098c9cde37d6516ee29894c2845a0a3
Oct 30 22:32:32.147375 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:32.147356    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf"
Oct 30 22:32:32.539521 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:32:32.539488    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8646992a_ed0b_4a46_b781_d9e3f5aff39b.slice/crio-f9bc9716fcca0a808487e12059acf8ffe3c6a24c8ada4e650b49d26e9bd4fc8e WatchSource:0}: Error finding container f9bc9716fcca0a808487e12059acf8ffe3c6a24c8ada4e650b49d26e9bd4fc8e: Status 404 returned error can't find the container with id f9bc9716fcca0a808487e12059acf8ffe3c6a24c8ada4e650b49d26e9bd4fc8e
Oct 30 22:32:32.539670 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:32.539638    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf"]
Oct 30 22:32:32.715767 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:32.715651    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:32.968285 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:32.968137    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l" event={"ID":"e0a34b6d-354a-4b67-8961-c336db52c10e","Type":"ContainerStarted","Data":"24ba921e10c332fde486d3024e564ea726a49eda70fe03c64f67f088e78f00ca"}
Oct 30 22:32:32.968285 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:32.968199    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l" event={"ID":"e0a34b6d-354a-4b67-8961-c336db52c10e","Type":"ContainerStarted","Data":"af54f9a911b73d52c5ad379ba049ed6c5098c9cde37d6516ee29894c2845a0a3"}
Oct 30 22:32:32.968285 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:32.968267    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l"
Oct 30 22:32:32.970939 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:32.970903    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf" event={"ID":"8646992a-ed0b-4a46-b781-d9e3f5aff39b","Type":"ContainerStarted","Data":"d44919212212e6415030333e16134280e7f917060b235da2a531a681888dbb8b"}
Oct 30 22:32:32.971102 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:32.970963    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf" event={"ID":"8646992a-ed0b-4a46-b781-d9e3f5aff39b","Type":"ContainerStarted","Data":"f9bc9716fcca0a808487e12059acf8ffe3c6a24c8ada4e650b49d26e9bd4fc8e"}
Oct 30 22:32:32.971270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:32.971212    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf"
Oct 30 22:32:32.971364 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:32.971290    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:32.972149 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:32.972132    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:32:33.005139 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:33.005072    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l" podStartSLOduration=2.005052723 podStartE2EDuration="2.005052723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:33.004093441 +0000 UTC m=+1538.360459756" watchObservedRunningTime="2026-10-30 22:32:33.005052723 +0000 UTC m=+1538.361419041"
Oct 30 22:32:33.087317 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:33.087247    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf" podStartSLOduration=2.087205572 podStartE2EDuration="2.087205572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:33.085077322 +0000 UTC m=+1538.441443637" watchObservedRunningTime="2026-10-30 22:32:33.087205572 +0000 UTC m=+1538.443571888"
Oct 30 22:32:33.653768 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:33.653732    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:33.836853 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:33.836804    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:34.851248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:34.846939    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:35.424169 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:35.424123    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:35.458637 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:35.458599    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq"]
Oct 30 22:32:35.465539 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:35.465510    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq"
Oct 30 22:32:35.470649 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:35.470619    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq"]
Oct 30 22:32:35.480183 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:35.480138    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6922d3\""
Oct 30 22:32:35.480429 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:35.480371    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6922d3-dockercfg-v5fn8\""
Oct 30 22:32:35.543253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:35.542549    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24e6bd5-4b2f-4060-b4cb-b2df03309b3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq\" (UID: \"c24e6bd5-4b2f-4060-b4cb-b2df03309b3a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq"
Oct 30 22:32:35.543253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:35.542684    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8p58\" (UniqueName: \"kubernetes.io/projected/c24e6bd5-4b2f-4060-b4cb-b2df03309b3a-kube-api-access-g8p58\") pod \"payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq\" (UID: \"c24e6bd5-4b2f-4060-b4cb-b2df03309b3a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq"
Oct 30 22:32:35.801550 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:35.798733    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24e6bd5-4b2f-4060-b4cb-b2df03309b3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq\" (UID: \"c24e6bd5-4b2f-4060-b4cb-b2df03309b3a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq"
Oct 30 22:32:35.801550 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:35.798868    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8p58\" (UniqueName: \"kubernetes.io/projected/c24e6bd5-4b2f-4060-b4cb-b2df03309b3a-kube-api-access-g8p58\") pod \"payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq\" (UID: \"c24e6bd5-4b2f-4060-b4cb-b2df03309b3a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq"
Oct 30 22:32:35.802774 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:35.802694    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24e6bd5-4b2f-4060-b4cb-b2df03309b3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq\" (UID: \"c24e6bd5-4b2f-4060-b4cb-b2df03309b3a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq"
Oct 30 22:32:35.808597 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:35.805558    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:35.904780 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:35.904703    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8p58\" (UniqueName: \"kubernetes.io/projected/c24e6bd5-4b2f-4060-b4cb-b2df03309b3a-kube-api-access-g8p58\") pod \"payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq\" (UID: \"c24e6bd5-4b2f-4060-b4cb-b2df03309b3a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq"
Oct 30 22:32:36.085778 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:36.085695    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq"
Oct 30 22:32:36.400467 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:36.400435    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq"]
Oct 30 22:32:37.017055 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:37.017013    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq" event={"ID":"c24e6bd5-4b2f-4060-b4cb-b2df03309b3a","Type":"ContainerStarted","Data":"75dccf023d12f60a21d74c5e905a437b983724c19129e41b7229fd55cf5b1e48"}
Oct 30 22:32:37.017540 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:37.017100    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq" event={"ID":"c24e6bd5-4b2f-4060-b4cb-b2df03309b3a","Type":"ContainerStarted","Data":"e1ac1d1bce1c785dc466b6e9ad9cf614daa1bf862146b95a6e4cbcfbae81c0f8"}
Oct 30 22:32:37.017540 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:37.017247    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq"
Oct 30 22:32:37.024688 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:37.024658    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:37.057338 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:37.057278    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq" podStartSLOduration=2.057259994 podStartE2EDuration="2.057259994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:37.052385641 +0000 UTC m=+1542.408751956" watchObservedRunningTime="2026-10-30 22:32:37.057259994 +0000 UTC m=+1542.413626311"
Oct 30 22:32:38.827571 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:38.827518    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:39.154445 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:39.154361    4889 patch_prober.go:29] interesting pod/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:32:39.154445 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:39.154427    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp" podUID="be263888-3047-4ee6-88cf-0fe7a033f3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:32:40.305470 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:32:40.305430    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:32:40.688571 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:40.688512    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:41.066111 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:41.066063    4889 generic.go:356] "Generic (PLEG): container finished" podID="be263888-3047-4ee6-88cf-0fe7a033f3e1" containerID="78ef6659fd1b429d3b13edf3a54e493019dfae74033c9ecb4d421143ba19d1b9" exitCode=0
Oct 30 22:32:41.066497 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:41.066143    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp" event={"ID":"be263888-3047-4ee6-88cf-0fe7a033f3e1","Type":"ContainerDied","Data":"78ef6659fd1b429d3b13edf3a54e493019dfae74033c9ecb4d421143ba19d1b9"}
Oct 30 22:32:41.066815 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:41.066786    4889 scope.go:122] "RemoveContainer" containerID="78ef6659fd1b429d3b13edf3a54e493019dfae74033c9ecb4d421143ba19d1b9"
Oct 30 22:32:41.068984 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:41.068952    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:32:41.069124 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:41.069027    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:32:41.069124 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:41.069062    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:32:42.078239 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:42.078193    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp" event={"ID":"be263888-3047-4ee6-88cf-0fe7a033f3e1","Type":"ContainerStarted","Data":"a030ba12d8f39b8aceeedc969b6fa04529aad4c194e41ad6ca4ba46b6ac026ec"}
Oct 30 22:32:42.521829 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:42.521789    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:42.667936 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:42.667893    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:32:42.668146 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:42.668120    4889 patch_prober.go:29] interesting pod/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:32:42.668209 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:42.668164    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp" podUID="be263888-3047-4ee6-88cf-0fe7a033f3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:32:43.388051 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:43.388015    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:43.568290 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:43.564006    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:43.669368 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:43.669283    4889 patch_prober.go:29] interesting pod/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:32:43.669368 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:43.669352    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp" podUID="be263888-3047-4ee6-88cf-0fe7a033f3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:32:44.042245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:44.016408    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf"
Oct 30 22:32:44.042245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:44.017537    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l"
Oct 30 22:32:44.670190 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:44.670151    4889 patch_prober.go:29] interesting pod/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:32:44.670654 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:44.670263    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp" podUID="be263888-3047-4ee6-88cf-0fe7a033f3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:32:44.746892 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:44.746853    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:45.668269 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:45.668236    4889 patch_prober.go:29] interesting pod/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:32:45.668467 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:45.668293    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp" podUID="be263888-3047-4ee6-88cf-0fe7a033f3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:32:46.502780 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:46.502744    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:46.668562 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:46.668519    4889 patch_prober.go:29] interesting pod/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:32:46.668743 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:46.668585    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp" podUID="be263888-3047-4ee6-88cf-0fe7a033f3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:32:47.667966 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:47.667930    4889 patch_prober.go:29] interesting pod/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:32:47.668465 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:47.667985    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp" podUID="be263888-3047-4ee6-88cf-0fe7a033f3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:32:47.966634 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:47.966530    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:48.041135 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:48.041102    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq"
Oct 30 22:32:48.380721 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:48.380685    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:48.555250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:48.549653    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:48.668351 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:48.668253    4889 patch_prober.go:29] interesting pod/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:32:48.668351 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:48.668310    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp" podUID="be263888-3047-4ee6-88cf-0fe7a033f3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:32:48.716137 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:48.716096    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:49.680289 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:49.680241    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:32:49.682587 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:49.682561    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:32:49.683529 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:49.683495    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:32:50.514248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:50.512761    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:52.248422 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:52.248386    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:54.033640 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:54.033605    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:55.599610 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:55.599571    4889 scope.go:122] "RemoveContainer" containerID="73a127efa766b7a42a4d634d676dc9b91d384681d01ab479f1af4721ac308b13"
Oct 30 22:32:56.236165 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:56.236118    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:58.297128 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:58.297095    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:59.733442 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:32:59.733382    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:33:00.580200 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:00.580092    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:02.714927 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:02.714872    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:04.629588 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:04.629546    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:06.472873 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:06.472825    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:06.835815 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:06.835725    4889 generic.go:356] "Generic (PLEG): container finished" podID="47a318ba-6fbd-479f-aa9a-e07dc2ab12a8" containerID="d73d742efb808e8ea43f9a3b9ae9c0e1a94061777bdd868b121bc998e8c61570" exitCode=0
Oct 30 22:33:06.835815 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:06.835803    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p" event={"ID":"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8","Type":"ContainerDied","Data":"d73d742efb808e8ea43f9a3b9ae9c0e1a94061777bdd868b121bc998e8c61570"}
Oct 30 22:33:06.836461 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:06.836441    4889 scope.go:122] "RemoveContainer" containerID="d73d742efb808e8ea43f9a3b9ae9c0e1a94061777bdd868b121bc998e8c61570"
Oct 30 22:33:06.838557 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:06.838531    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:33:06.838624 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:06.838605    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:33:06.838660 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:06.838640    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:33:07.847291 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:07.847254    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p" event={"ID":"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8","Type":"ContainerStarted","Data":"01918c94a8f5ead28c67206e949821698055c2d2b0be5203dc712757e723491a"}
Oct 30 22:33:07.932334 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:07.931068    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:08.240048 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:08.239956    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:08.495887 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:08.495853    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:33:08.497037 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:08.497008    4889 patch_prober.go:29] interesting pod/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 22:33:08.497173 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:08.497078    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p" podUID="47a318ba-6fbd-479f-aa9a-e07dc2ab12a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 22:33:09.008415 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:09.008383    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:33:09.496504 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:09.496466    4889 patch_prober.go:29] interesting pod/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 22:33:09.496710 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:09.496552    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p" podUID="47a318ba-6fbd-479f-aa9a-e07dc2ab12a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 22:33:10.149422 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:10.149380    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:10.307210 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:33:10.307160    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:33:10.496655 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:10.496622    4889 patch_prober.go:29] interesting pod/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 22:33:10.496836 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:10.496683    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p" podUID="47a318ba-6fbd-479f-aa9a-e07dc2ab12a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 22:33:11.496052 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:11.496016    4889 patch_prober.go:29] interesting pod/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 22:33:11.496504 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:11.496067    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p" podUID="47a318ba-6fbd-479f-aa9a-e07dc2ab12a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 22:33:12.293250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:12.288565    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:12.496243 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:12.496198    4889 patch_prober.go:29] interesting pod/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 22:33:12.496767 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:12.496293    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p" podUID="47a318ba-6fbd-479f-aa9a-e07dc2ab12a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 22:33:13.501895 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:13.501857    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:33:13.503041 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:13.503018    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:33:14.228442 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:14.228395    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:14.393876 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:14.393839    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:14.567240 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:14.567192    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:16.088766 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:16.088723    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:17.885251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:17.879731    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:19.205747 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:19.205715    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm"]
Oct 30 22:33:19.369189 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:19.369145    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:19.542632 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:19.542603    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:19.711362 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:19.711323    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:19.922201 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:19.922116    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:21.738959 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:21.738922    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:23.634779 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:23.634744    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:24.259953 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:24.259645    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm" podUID="6f7253a4-2b57-4ac8-b8ee-11da584e0e64" containerName="payload-pre-processing" containerID="cri-o://d1622b8e416d1a4e0fda40b01edd3b047eb8de380a3461a7caa7d3f194f3cd2e" gracePeriod=25
Oct 30 22:33:24.606864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:24.606838    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm"
Oct 30 22:33:24.748792 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:24.748754    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f7253a4-2b57-4ac8-b8ee-11da584e0e64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7253a4-2b57-4ac8-b8ee-11da584e0e64-plugins-config-volume\") pod \"6f7253a4-2b57-4ac8-b8ee-11da584e0e64\" (UID: \"6f7253a4-2b57-4ac8-b8ee-11da584e0e64\") "
Oct 30 22:33:24.749342 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:24.748819    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f7253a4-2b57-4ac8-b8ee-11da584e0e64-kube-api-access-mmcdt\" (UniqueName: \"kubernetes.io/projected/6f7253a4-2b57-4ac8-b8ee-11da584e0e64-kube-api-access-mmcdt\") pod \"6f7253a4-2b57-4ac8-b8ee-11da584e0e64\" (UID: \"6f7253a4-2b57-4ac8-b8ee-11da584e0e64\") "
Oct 30 22:33:24.749342 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:24.749213    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f7253a4-2b57-4ac8-b8ee-11da584e0e64-plugins-config-volume" pod "6f7253a4-2b57-4ac8-b8ee-11da584e0e64" (UID: "6f7253a4-2b57-4ac8-b8ee-11da584e0e64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:24.749604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:24.749582    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7253a4-2b57-4ac8-b8ee-11da584e0e64-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:24.751201 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:24.751174    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7253a4-2b57-4ac8-b8ee-11da584e0e64-kube-api-access-mmcdt" pod "6f7253a4-2b57-4ac8-b8ee-11da584e0e64" (UID: "6f7253a4-2b57-4ac8-b8ee-11da584e0e64"). InnerVolumeSpecName "kube-api-access-mmcdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:24.851873 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:24.851789    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmcdt\" (UniqueName: \"kubernetes.io/projected/6f7253a4-2b57-4ac8-b8ee-11da584e0e64-kube-api-access-mmcdt\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:25.343900 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:25.343861    4889 generic.go:356] "Generic (PLEG): container finished" podID="6f7253a4-2b57-4ac8-b8ee-11da584e0e64" containerID="d1622b8e416d1a4e0fda40b01edd3b047eb8de380a3461a7caa7d3f194f3cd2e" exitCode=0
Oct 30 22:33:25.343900 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:25.343907    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm"
Oct 30 22:33:25.344194 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:25.343949    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm" event={"ID":"6f7253a4-2b57-4ac8-b8ee-11da584e0e64","Type":"ContainerDied","Data":"d1622b8e416d1a4e0fda40b01edd3b047eb8de380a3461a7caa7d3f194f3cd2e"}
Oct 30 22:33:25.344194 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:25.344011    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm" event={"ID":"6f7253a4-2b57-4ac8-b8ee-11da584e0e64","Type":"ContainerDied","Data":"e7dba7b91d42980b4299cf14191aacfbdbfb58d4ddb2e2ffabffed7ec02b4cdd"}
Oct 30 22:33:25.344194 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:25.344047    4889 scope.go:122] "RemoveContainer" containerID="d1622b8e416d1a4e0fda40b01edd3b047eb8de380a3461a7caa7d3f194f3cd2e"
Oct 30 22:33:25.356493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:25.356474    4889 scope.go:122] "RemoveContainer" containerID="d1622b8e416d1a4e0fda40b01edd3b047eb8de380a3461a7caa7d3f194f3cd2e"
Oct 30 22:33:25.356864 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:33:25.356845    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1622b8e416d1a4e0fda40b01edd3b047eb8de380a3461a7caa7d3f194f3cd2e\": container with ID starting with d1622b8e416d1a4e0fda40b01edd3b047eb8de380a3461a7caa7d3f194f3cd2e not found: ID does not exist" containerID="d1622b8e416d1a4e0fda40b01edd3b047eb8de380a3461a7caa7d3f194f3cd2e"
Oct 30 22:33:25.356935 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:25.356876    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1622b8e416d1a4e0fda40b01edd3b047eb8de380a3461a7caa7d3f194f3cd2e"} err="failed to get container status \"d1622b8e416d1a4e0fda40b01edd3b047eb8de380a3461a7caa7d3f194f3cd2e\": rpc error: code = NotFound desc = could not find container \"d1622b8e416d1a4e0fda40b01edd3b047eb8de380a3461a7caa7d3f194f3cd2e\": container with ID starting with d1622b8e416d1a4e0fda40b01edd3b047eb8de380a3461a7caa7d3f194f3cd2e not found: ID does not exist"
Oct 30 22:33:25.394522 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:25.394496    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm"]
Oct 30 22:33:25.405514 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:25.405484    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-de31dd-7579cc4f79-rq4dm"]
Oct 30 22:33:25.536248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:25.534806    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:26.778428 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:26.778396    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f7253a4-2b57-4ac8-b8ee-11da584e0e64" path="/var/lib/kubelet/pods/6f7253a4-2b57-4ac8-b8ee-11da584e0e64/volumes"
Oct 30 22:33:27.373247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:27.371598    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:29.268379 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:29.268336    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:31.069373 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:31.063023    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:32.714198 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:32.714158    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:34.123050 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:34.123015    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx"]
Oct 30 22:33:34.124154 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:34.124119    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx" podUID="d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35" containerName="maas-api" containerID="cri-o://605853fcb101e97205a0660ede147a193138a510b44356204783b729e9b5f37a" gracePeriod=30
Oct 30 22:33:34.247866 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:34.247824    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:34.796597 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:34.796499    4889 generic.go:356] "Generic (PLEG): container finished" podID="d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35" containerID="605853fcb101e97205a0660ede147a193138a510b44356204783b729e9b5f37a" exitCode=0
Oct 30 22:33:34.872750 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:34.864032    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx" event={"ID":"d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35","Type":"ContainerDied","Data":"605853fcb101e97205a0660ede147a193138a510b44356204783b729e9b5f37a"}
Oct 30 22:33:34.941252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:34.921612    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx"
Oct 30 22:33:35.017173 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:35.015590    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-openshift-service-ca\") pod \"d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35\" (UID: \"d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35\") "
Oct 30 22:33:35.017173 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:35.015647    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-maas-api-tls\") pod \"d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35\" (UID: \"d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35\") "
Oct 30 22:33:35.017173 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:35.015682    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-metrics-certs\") pod \"d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35\" (UID: \"d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35\") "
Oct 30 22:33:35.017173 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:35.015809    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-kube-api-access-bf4pt\" (UniqueName: \"kubernetes.io/projected/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-kube-api-access-bf4pt\") pod \"d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35\" (UID: \"d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35\") "
Oct 30 22:33:35.047423 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:35.046885    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-metrics-certs" pod "d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35" (UID: "d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:35.047423 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:35.047051    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-kube-api-access-bf4pt" pod "d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35" (UID: "d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35"). InnerVolumeSpecName "kube-api-access-bf4pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:35.048123 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:35.048084    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-openshift-service-ca" pod "d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35" (UID: "d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:35.049876 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:35.049834    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-maas-api-tls" pod "d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35" (UID: "d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:35.100733 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:35.100621    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:35.118690 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:35.118651    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:35.118690 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:35.118687    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:35.118690 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:35.118698    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:35.119004 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:35.118713    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf4pt\" (UniqueName: \"kubernetes.io/projected/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35-kube-api-access-bf4pt\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:35.280453 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:35.280402    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:35.474300 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:35.468732    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:35.809716 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:35.809687    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx" event={"ID":"d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35","Type":"ContainerDied","Data":"3133c51f38a0b036e1ef6d5b715916fc075c0166781620af7a31adc31c93061d"}
Oct 30 22:33:35.809716 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:35.809713    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx"
Oct 30 22:33:35.809950 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:35.809751    4889 scope.go:122] "RemoveContainer" containerID="605853fcb101e97205a0660ede147a193138a510b44356204783b729e9b5f37a"
Oct 30 22:33:35.844399 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:35.844363    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:35.904311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:35.904278    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx"]
Oct 30 22:33:35.914961 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:35.914924    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx"]
Oct 30 22:33:36.778958 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:36.778922    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35" path="/var/lib/kubelet/pods/d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35/volumes"
Oct 30 22:33:36.821717 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:36.821687    4889 generic.go:356] "Generic (PLEG): container finished" podID="41d10d03-c9af-4c80-b5c5-692d1a3953af" containerID="4135272e3e3db1908e8789c1be14aff438670105bfaebf377c19742d1df753d3" exitCode=0
Oct 30 22:33:36.821913 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:36.821764    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj" event={"ID":"41d10d03-c9af-4c80-b5c5-692d1a3953af","Type":"ContainerDied","Data":"4135272e3e3db1908e8789c1be14aff438670105bfaebf377c19742d1df753d3"}
Oct 30 22:33:36.822443 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:36.822418    4889 scope.go:122] "RemoveContainer" containerID="4135272e3e3db1908e8789c1be14aff438670105bfaebf377c19742d1df753d3"
Oct 30 22:33:36.824783 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:36.824752    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:33:36.824880 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:36.824839    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:33:36.824927 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:36.824901    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:33:37.393351 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:37.393317    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:37.595249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:37.594521    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:37.621248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:37.619452    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-370ec2-6f6f65cdc7-g9rhx" podUID="d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.129:8443/health\": context deadline exceeded"
Oct 30 22:33:37.754736 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:37.754696    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"]
Oct 30 22:33:37.756178 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:37.756148    4889 cpu_manager.go:414] "containerMap: removing container" podUID="d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35" containerName="maas-api"
Oct 30 22:33:37.756178 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:37.756176    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35" containerName="maas-api"
Oct 30 22:33:37.756415 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:37.756191    4889 cpu_manager.go:414] "containerMap: removing container" podUID="6f7253a4-2b57-4ac8-b8ee-11da584e0e64" containerName="payload-pre-processing"
Oct 30 22:33:37.756415 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:37.756200    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f7253a4-2b57-4ac8-b8ee-11da584e0e64" containerName="payload-pre-processing"
Oct 30 22:33:37.756528 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:37.756442    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2a82e58-ed4d-48a2-ae3b-ff1a16f2aa35" containerName="maas-api"
Oct 30 22:33:37.756528 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:37.756457    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f7253a4-2b57-4ac8-b8ee-11da584e0e64" containerName="payload-pre-processing"
Oct 30 22:33:37.760842 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:37.760812    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:37.779208 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:37.779177    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"]
Oct 30 22:33:37.781469 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:37.781447    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-bdd94b-openshift-default-dockercfg-cpgxw\""
Oct 30 22:33:37.834874 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:37.834837    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj" event={"ID":"41d10d03-c9af-4c80-b5c5-692d1a3953af","Type":"ContainerStarted","Data":"89f6e908963789f1463396659e44881b9816d9f25db9e9d1fb22e47b7a3b1884"}
Oct 30 22:33:37.877952 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:37.877917    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:37.897306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:37.897269    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-credential-socket\") pod \"e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") " pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:37.897306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:37.897312    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-workload-certs\") pod \"e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") " pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:37.897564 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:37.897373    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-token\") pod \"e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") " pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:37.897622 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:37.897598    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-data\") pod \"e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") " pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:37.897891 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:37.897869    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-podinfo\") pod \"e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") " pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:37.898331 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:37.898301    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istiod-ca-cert\") pod \"e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") " pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:37.898461 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:37.898422    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-envoy\") pod \"e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") " pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:37.898578 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:37.898556    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-workload-socket\") pod \"e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") " pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:37.898684 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:37.898615    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c68d\" (UniqueName: \"kubernetes.io/projected/55f78a17-ccc6-4f75-902e-e048a59e6c6a-kube-api-access-9c68d\") pod \"e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") " pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:38.011301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:38.003822    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istiod-ca-cert\") pod \"e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") " pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:38.011301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:38.003880    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-envoy\") pod \"e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") " pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:38.011301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:38.003983    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-workload-socket\") pod \"e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") " pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:38.011301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:38.004041    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c68d\" (UniqueName: \"kubernetes.io/projected/55f78a17-ccc6-4f75-902e-e048a59e6c6a-kube-api-access-9c68d\") pod \"e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") " pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:38.011301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:38.004079    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-credential-socket\") pod \"e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") " pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:38.011301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:38.004113    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-workload-certs\") pod \"e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") " pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:38.011301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:38.004138    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-token\") pod \"e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") " pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:38.011301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:38.004177    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-data\") pod \"e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") " pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:38.011301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:38.004366    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-podinfo\") pod \"e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") " pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:38.011301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:38.007093    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istiod-ca-cert\") pod \"e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") " pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:38.011301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:38.007384    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-workload-certs\") pod \"e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") " pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:38.011301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:38.007639    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-credential-socket\") pod \"e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") " pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:38.011301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:38.008204    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-data\") pod \"e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") " pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:38.011301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:38.008628    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-workload-socket\") pod \"e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") " pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:38.041257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:38.031919    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-podinfo\") pod \"e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") " pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:38.041257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:38.039757    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-token\") pod \"e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") " pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:38.088886 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:38.086868    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:38.107252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:38.104542    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-envoy\") pod \"e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") " pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:38.122256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:38.117364    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c68d\" (UniqueName: \"kubernetes.io/projected/55f78a17-ccc6-4f75-902e-e048a59e6c6a-kube-api-access-9c68d\") pod \"e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") " pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:38.394066 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:38.390986    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:38.804248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:38.803884    4889 patch_prober.go:29] interesting pod/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 22:33:38.804248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:38.803941    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj" podUID="41d10d03-c9af-4c80-b5c5-692d1a3953af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 22:33:39.246249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:39.245969    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:39.271561 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:39.271509    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:33:39.271961 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:39.271928    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:33:39.272165 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:39.272141    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:33:39.283180 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:39.283137    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:33:39.283495 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:39.283480    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"]
Oct 30 22:33:39.779267 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:39.779214    4889 patch_prober.go:29] interesting pod/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 22:33:39.779484 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:39.779284    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj" podUID="41d10d03-c9af-4c80-b5c5-692d1a3953af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 22:33:40.080119 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:40.078030    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:40.286320 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:40.286283    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p" event={"ID":"55f78a17-ccc6-4f75-902e-e048a59e6c6a","Type":"ContainerStarted","Data":"3147bbc32cb944f07e6d215f8d6c35dc3cabd7134ec9f53e91b25b94fd68bf26"}
Oct 30 22:33:40.286494 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:40.286342    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p" event={"ID":"55f78a17-ccc6-4f75-902e-e048a59e6c6a","Type":"ContainerStarted","Data":"7edf6ec737ab614e8d39bcff3ed582c58513e738fe9e32f8b778c60d3beca356"}
Oct 30 22:33:40.305343 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:33:40.305304    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:33:40.340137 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:40.340030    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p" podStartSLOduration=3.34001433 podStartE2EDuration="3.34001433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:40.326722082 +0000 UTC m=+1605.683088399" watchObservedRunningTime="2026-10-30 22:33:40.34001433 +0000 UTC m=+1605.696380644"
Oct 30 22:33:40.392993 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:40.392958    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:40.399975 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:40.399946    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:40.778532 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:40.778498    4889 patch_prober.go:29] interesting pod/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 22:33:40.778678 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:40.778547    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj" podUID="41d10d03-c9af-4c80-b5c5-692d1a3953af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 22:33:41.295004 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:41.294976    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:41.296159 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:41.296138    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:33:41.779359 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:41.779314    4889 patch_prober.go:29] interesting pod/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 22:33:41.779565 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:41.779371    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj" podUID="41d10d03-c9af-4c80-b5c5-692d1a3953af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 22:33:42.162676 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:42.161587    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:42.789442 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:42.789403    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:42.809886 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:42.809836    4889 patch_prober.go:29] interesting pod/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 22:33:42.810176 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:42.810146    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj" podUID="41d10d03-c9af-4c80-b5c5-692d1a3953af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 22:33:42.973677 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:42.973640    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:43.001372 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:43.001331    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"]
Oct 30 22:33:43.002200 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:43.001915    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95" podUID="03297afe-0b96-48ea-bcb8-849f2cd95bb1" containerName="istio-proxy" containerID="cri-o://09ff567bd01776e200fe9ec7272caa5ef2af745275e006d00cc853fd1206c436" gracePeriod=30
Oct 30 22:33:43.044705 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:43.044615    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"]
Oct 30 22:33:43.045504 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:43.045481    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh" podUID="29631b37-3676-4395-a55d-f09f9b3e0025" containerName="istio-proxy" containerID="cri-o://81a7d52cdd909cd00bcba9e276b378ab0baeb3a27f535e841886401174e2459b" gracePeriod=30
Oct 30 22:33:43.053695 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:43.053665    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"]
Oct 30 22:33:43.053917 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:43.053883    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp" podUID="35eb0630-0a80-41fc-ab11-6d2577213921" containerName="istio-proxy" containerID="cri-o://7a75e86e8983ed0303c752dc60bd99135a363b39c1bdccb7b47715d8702d5710" gracePeriod=30
Oct 30 22:33:43.059518 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:43.059494    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"]
Oct 30 22:33:43.059909 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:43.059883    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t" podUID="cf853efa-c753-4ce6-834d-c315fba26177" containerName="istio-proxy" containerID="cri-o://0e6e14c1e558127bda6e7dde7960e9a93c3dab845c6761f24e46a63b131307ab" gracePeriod=30
Oct 30 22:33:43.193914 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:43.193877    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:43.778585 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:43.778553    4889 patch_prober.go:29] interesting pod/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 22:33:43.778806 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:43.778610    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj" podUID="41d10d03-c9af-4c80-b5c5-692d1a3953af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 22:33:44.400290 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:44.388988    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:44.697935 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:44.697833    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:44.784153 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:44.784126    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:33:44.784711 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:44.784695    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:33:44.785423 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:44.785407    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:33:44.967302 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:44.925836    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:46.543547 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:46.543488    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:48.035586 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.035524    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:48.293147 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.293100    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:48.391551 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.391516    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:33:48.479474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.473007    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"
Oct 30 22:33:48.479474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.473014    4889 generic.go:356] "Generic (PLEG): container finished" podID="03297afe-0b96-48ea-bcb8-849f2cd95bb1" containerID="09ff567bd01776e200fe9ec7272caa5ef2af745275e006d00cc853fd1206c436" exitCode=0
Oct 30 22:33:48.479474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.473150    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95" event={"ID":"03297afe-0b96-48ea-bcb8-849f2cd95bb1","Type":"ContainerDied","Data":"09ff567bd01776e200fe9ec7272caa5ef2af745275e006d00cc853fd1206c436"}
Oct 30 22:33:48.479474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.473193    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95" event={"ID":"03297afe-0b96-48ea-bcb8-849f2cd95bb1","Type":"ContainerDied","Data":"26a3fe67c5e8df9d3a7c093afe6d9458bd0e958354074df5e879d7e7e2afdd23"}
Oct 30 22:33:48.479474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.473218    4889 scope.go:122] "RemoveContainer" containerID="09ff567bd01776e200fe9ec7272caa5ef2af745275e006d00cc853fd1206c436"
Oct 30 22:33:48.479474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.477804    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istiod-ca-cert" pod "03297afe-0b96-48ea-bcb8-849f2cd95bb1" (UID: "03297afe-0b96-48ea-bcb8-849f2cd95bb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:48.479474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.477299    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istiod-ca-cert\") pod \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") "
Oct 30 22:33:48.479474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.477950    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03297afe-0b96-48ea-bcb8-849f2cd95bb1-kube-api-access-h5fkc\" (UniqueName: \"kubernetes.io/projected/03297afe-0b96-48ea-bcb8-849f2cd95bb1-kube-api-access-h5fkc\") pod \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") "
Oct 30 22:33:48.479474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.478243    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-data\") pod \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") "
Oct 30 22:33:48.479474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.478304    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-credential-socket\") pod \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") "
Oct 30 22:33:48.479474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.478437    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-workload-certs\") pod \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") "
Oct 30 22:33:48.479474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.478490    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-podinfo\") pod \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") "
Oct 30 22:33:48.479474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.478536    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-workload-socket\") pod \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") "
Oct 30 22:33:48.479474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.478638    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-envoy\") pod \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") "
Oct 30 22:33:48.479474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.478715    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-token\" (UniqueName: \"kubernetes.io/projected/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-token\") pod \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\" (UID: \"03297afe-0b96-48ea-bcb8-849f2cd95bb1\") "
Oct 30 22:33:48.484321 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.479670    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-workload-certs" pod "03297afe-0b96-48ea-bcb8-849f2cd95bb1" (UID: "03297afe-0b96-48ea-bcb8-849f2cd95bb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.484321 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.479903    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-credential-socket" pod "03297afe-0b96-48ea-bcb8-849f2cd95bb1" (UID: "03297afe-0b96-48ea-bcb8-849f2cd95bb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.484321 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.481029    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-data" pod "03297afe-0b96-48ea-bcb8-849f2cd95bb1" (UID: "03297afe-0b96-48ea-bcb8-849f2cd95bb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.500915 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.500884    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-workload-socket" pod "03297afe-0b96-48ea-bcb8-849f2cd95bb1" (UID: "03297afe-0b96-48ea-bcb8-849f2cd95bb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.508468 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.504201    4889 generic.go:356] "Generic (PLEG): container finished" podID="cf853efa-c753-4ce6-834d-c315fba26177" containerID="0e6e14c1e558127bda6e7dde7960e9a93c3dab845c6761f24e46a63b131307ab" exitCode=0
Oct 30 22:33:48.508468 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.504466    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t" event={"ID":"cf853efa-c753-4ce6-834d-c315fba26177","Type":"ContainerDied","Data":"0e6e14c1e558127bda6e7dde7960e9a93c3dab845c6761f24e46a63b131307ab"}
Oct 30 22:33:48.508468 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.507060    4889 generic.go:356] "Generic (PLEG): container finished" podID="29631b37-3676-4395-a55d-f09f9b3e0025" containerID="81a7d52cdd909cd00bcba9e276b378ab0baeb3a27f535e841886401174e2459b" exitCode=0
Oct 30 22:33:48.508468 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.507162    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh" event={"ID":"29631b37-3676-4395-a55d-f09f9b3e0025","Type":"ContainerDied","Data":"81a7d52cdd909cd00bcba9e276b378ab0baeb3a27f535e841886401174e2459b"}
Oct 30 22:33:48.509094 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.509059    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-envoy" pod "03297afe-0b96-48ea-bcb8-849f2cd95bb1" (UID: "03297afe-0b96-48ea-bcb8-849f2cd95bb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.509762 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.509731    4889 generic.go:356] "Generic (PLEG): container finished" podID="35eb0630-0a80-41fc-ab11-6d2577213921" containerID="7a75e86e8983ed0303c752dc60bd99135a363b39c1bdccb7b47715d8702d5710" exitCode=0
Oct 30 22:33:48.509883 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.509805    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp" event={"ID":"35eb0630-0a80-41fc-ab11-6d2577213921","Type":"ContainerDied","Data":"7a75e86e8983ed0303c752dc60bd99135a363b39c1bdccb7b47715d8702d5710"}
Oct 30 22:33:48.511137 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.511107    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.511137 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.511138    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.511317 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.511152    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.511317 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.511166    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.511317 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.511179    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.511317 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.511203    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.511627 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.511398    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-token" pod "03297afe-0b96-48ea-bcb8-849f2cd95bb1" (UID: "03297afe-0b96-48ea-bcb8-849f2cd95bb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:48.512317 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.512281    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-podinfo" pod "03297afe-0b96-48ea-bcb8-849f2cd95bb1" (UID: "03297afe-0b96-48ea-bcb8-849f2cd95bb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:33:48.513214 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.513184    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03297afe-0b96-48ea-bcb8-849f2cd95bb1-kube-api-access-h5fkc" pod "03297afe-0b96-48ea-bcb8-849f2cd95bb1" (UID: "03297afe-0b96-48ea-bcb8-849f2cd95bb1"). InnerVolumeSpecName "kube-api-access-h5fkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:48.531060 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.531027    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:48.534730 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.534707    4889 scope.go:122] "RemoveContainer" containerID="09ff567bd01776e200fe9ec7272caa5ef2af745275e006d00cc853fd1206c436"
Oct 30 22:33:48.536260 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:33:48.536084    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09ff567bd01776e200fe9ec7272caa5ef2af745275e006d00cc853fd1206c436\": container with ID starting with 09ff567bd01776e200fe9ec7272caa5ef2af745275e006d00cc853fd1206c436 not found: ID does not exist" containerID="09ff567bd01776e200fe9ec7272caa5ef2af745275e006d00cc853fd1206c436"
Oct 30 22:33:48.536260 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.536123    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09ff567bd01776e200fe9ec7272caa5ef2af745275e006d00cc853fd1206c436"} err="failed to get container status \"09ff567bd01776e200fe9ec7272caa5ef2af745275e006d00cc853fd1206c436\": rpc error: code = NotFound desc = could not find container \"09ff567bd01776e200fe9ec7272caa5ef2af745275e006d00cc853fd1206c436\": container with ID starting with 09ff567bd01776e200fe9ec7272caa5ef2af745275e006d00cc853fd1206c436 not found: ID does not exist"
Oct 30 22:33:48.536260 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.536167    4889 scope.go:122] "RemoveContainer" containerID="e6213e7a04157ee84a7d9dc781c8bc7e201caac96e85d397e91060fa41908c57"
Oct 30 22:33:48.538403 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.537295    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:33:48.541997 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.541978    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:33:48.613613 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.612386    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29631b37-3676-4395-a55d-f09f9b3e0025-kube-api-access-lhplt\" (UniqueName: \"kubernetes.io/projected/29631b37-3676-4395-a55d-f09f9b3e0025-kube-api-access-lhplt\") pod \"29631b37-3676-4395-a55d-f09f9b3e0025\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") "
Oct 30 22:33:48.613613 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.612451    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35eb0630-0a80-41fc-ab11-6d2577213921-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35eb0630-0a80-41fc-ab11-6d2577213921-istiod-ca-cert\") pod \"35eb0630-0a80-41fc-ab11-6d2577213921\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") "
Oct 30 22:33:48.613613 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.612480    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29631b37-3676-4395-a55d-f09f9b3e0025-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29631b37-3676-4395-a55d-f09f9b3e0025-istiod-ca-cert\") pod \"29631b37-3676-4395-a55d-f09f9b3e0025\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") "
Oct 30 22:33:48.613613 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.612513    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-credential-socket\") pod \"29631b37-3676-4395-a55d-f09f9b3e0025\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") "
Oct 30 22:33:48.613613 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.612570    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29631b37-3676-4395-a55d-f09f9b3e0025-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29631b37-3676-4395-a55d-f09f9b3e0025-istio-podinfo\") pod \"29631b37-3676-4395-a55d-f09f9b3e0025\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") "
Oct 30 22:33:48.613613 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.612624    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-workload-certs\") pod \"35eb0630-0a80-41fc-ab11-6d2577213921\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") "
Oct 30 22:33:48.613613 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.612642    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-workload-socket\") pod \"35eb0630-0a80-41fc-ab11-6d2577213921\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") "
Oct 30 22:33:48.613613 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.612683    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-credential-socket\") pod \"35eb0630-0a80-41fc-ab11-6d2577213921\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") "
Oct 30 22:33:48.613613 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.612711    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35eb0630-0a80-41fc-ab11-6d2577213921-kube-api-access-mhxbq\" (UniqueName: \"kubernetes.io/projected/35eb0630-0a80-41fc-ab11-6d2577213921-kube-api-access-mhxbq\") pod \"35eb0630-0a80-41fc-ab11-6d2577213921\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") "
Oct 30 22:33:48.613613 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.612777    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-workload-socket\") pod \"29631b37-3676-4395-a55d-f09f9b3e0025\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") "
Oct 30 22:33:48.613613 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.612793    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35eb0630-0a80-41fc-ab11-6d2577213921-istio-token\" (UniqueName: \"kubernetes.io/projected/35eb0630-0a80-41fc-ab11-6d2577213921-istio-token\") pod \"35eb0630-0a80-41fc-ab11-6d2577213921\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") "
Oct 30 22:33:48.613613 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.612822    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-istio-data\") pod \"35eb0630-0a80-41fc-ab11-6d2577213921\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") "
Oct 30 22:33:48.613613 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.612845    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29631b37-3676-4395-a55d-f09f9b3e0025-istio-token\" (UniqueName: \"kubernetes.io/projected/29631b37-3676-4395-a55d-f09f9b3e0025-istio-token\") pod \"29631b37-3676-4395-a55d-f09f9b3e0025\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") "
Oct 30 22:33:48.613613 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.612877    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-workload-certs\") pod \"29631b37-3676-4395-a55d-f09f9b3e0025\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") "
Oct 30 22:33:48.613613 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.612926    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35eb0630-0a80-41fc-ab11-6d2577213921-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35eb0630-0a80-41fc-ab11-6d2577213921-istio-podinfo\") pod \"35eb0630-0a80-41fc-ab11-6d2577213921\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") "
Oct 30 22:33:48.613613 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.613009    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-istio-envoy\") pod \"35eb0630-0a80-41fc-ab11-6d2577213921\" (UID: \"35eb0630-0a80-41fc-ab11-6d2577213921\") "
Oct 30 22:33:48.613613 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.613033    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-istio-envoy\") pod \"29631b37-3676-4395-a55d-f09f9b3e0025\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") "
Oct 30 22:33:48.613613 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.613066    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-istio-data\") pod \"29631b37-3676-4395-a55d-f09f9b3e0025\" (UID: \"29631b37-3676-4395-a55d-f09f9b3e0025\") "
Oct 30 22:33:48.614785 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.613759    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-workload-socket" pod "29631b37-3676-4395-a55d-f09f9b3e0025" (UID: "29631b37-3676-4395-a55d-f09f9b3e0025"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.614785 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.613992    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-istio-data" pod "35eb0630-0a80-41fc-ab11-6d2577213921" (UID: "35eb0630-0a80-41fc-ab11-6d2577213921"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.615194 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.615169    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-workload-certs" pod "29631b37-3676-4395-a55d-f09f9b3e0025" (UID: "29631b37-3676-4395-a55d-f09f9b3e0025"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.615567 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.615546    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-credential-socket" pod "29631b37-3676-4395-a55d-f09f9b3e0025" (UID: "29631b37-3676-4395-a55d-f09f9b3e0025"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.616360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.616136    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35eb0630-0a80-41fc-ab11-6d2577213921-istiod-ca-cert" pod "35eb0630-0a80-41fc-ab11-6d2577213921" (UID: "35eb0630-0a80-41fc-ab11-6d2577213921"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:48.616787 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.616595    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29631b37-3676-4395-a55d-f09f9b3e0025-istiod-ca-cert" pod "29631b37-3676-4395-a55d-f09f9b3e0025" (UID: "29631b37-3676-4395-a55d-f09f9b3e0025"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:48.626273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.617039    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35eb0630-0a80-41fc-ab11-6d2577213921-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.626273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.617063    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29631b37-3676-4395-a55d-f09f9b3e0025-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.626273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.617064    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-istio-data" pod "29631b37-3676-4395-a55d-f09f9b3e0025" (UID: "29631b37-3676-4395-a55d-f09f9b3e0025"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.626273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.617076    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.626273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.617110    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.626273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.617132    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.626273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.617150    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.626273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.617163    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.626273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.617176    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03297afe-0b96-48ea-bcb8-849f2cd95bb1-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.626273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.617174    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35eb0630-0a80-41fc-ab11-6d2577213921-istio-podinfo" pod "35eb0630-0a80-41fc-ab11-6d2577213921" (UID: "35eb0630-0a80-41fc-ab11-6d2577213921"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:33:48.626273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.617190    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5fkc\" (UniqueName: \"kubernetes.io/projected/03297afe-0b96-48ea-bcb8-849f2cd95bb1-kube-api-access-h5fkc\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.626273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.617445    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-workload-socket" pod "35eb0630-0a80-41fc-ab11-6d2577213921" (UID: "35eb0630-0a80-41fc-ab11-6d2577213921"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.626273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.617624    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-credential-socket" pod "35eb0630-0a80-41fc-ab11-6d2577213921" (UID: "35eb0630-0a80-41fc-ab11-6d2577213921"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.626273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.617680    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-istio-envoy" pod "35eb0630-0a80-41fc-ab11-6d2577213921" (UID: "35eb0630-0a80-41fc-ab11-6d2577213921"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.626273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.619637    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-istio-envoy" pod "29631b37-3676-4395-a55d-f09f9b3e0025" (UID: "29631b37-3676-4395-a55d-f09f9b3e0025"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.626273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.619686    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29631b37-3676-4395-a55d-f09f9b3e0025-kube-api-access-lhplt" pod "29631b37-3676-4395-a55d-f09f9b3e0025" (UID: "29631b37-3676-4395-a55d-f09f9b3e0025"). InnerVolumeSpecName "kube-api-access-lhplt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:48.626273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.619708    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35eb0630-0a80-41fc-ab11-6d2577213921-istio-token" pod "35eb0630-0a80-41fc-ab11-6d2577213921" (UID: "35eb0630-0a80-41fc-ab11-6d2577213921"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:48.626273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.621164    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29631b37-3676-4395-a55d-f09f9b3e0025-istio-podinfo" pod "29631b37-3676-4395-a55d-f09f9b3e0025" (UID: "29631b37-3676-4395-a55d-f09f9b3e0025"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:33:48.626273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.621198    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35eb0630-0a80-41fc-ab11-6d2577213921-kube-api-access-mhxbq" pod "35eb0630-0a80-41fc-ab11-6d2577213921" (UID: "35eb0630-0a80-41fc-ab11-6d2577213921"). InnerVolumeSpecName "kube-api-access-mhxbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:48.626273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.621553    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29631b37-3676-4395-a55d-f09f9b3e0025-istio-token" pod "29631b37-3676-4395-a55d-f09f9b3e0025" (UID: "29631b37-3676-4395-a55d-f09f9b3e0025"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:48.626273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.625454    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-workload-certs" pod "35eb0630-0a80-41fc-ab11-6d2577213921" (UID: "35eb0630-0a80-41fc-ab11-6d2577213921"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.657980 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.657946    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:33:48.718686 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.718653    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-workload-socket\") pod \"cf853efa-c753-4ce6-834d-c315fba26177\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") "
Oct 30 22:33:48.718858 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.718712    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf853efa-c753-4ce6-834d-c315fba26177-kube-api-access-nwxjq\" (UniqueName: \"kubernetes.io/projected/cf853efa-c753-4ce6-834d-c315fba26177-kube-api-access-nwxjq\") pod \"cf853efa-c753-4ce6-834d-c315fba26177\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") "
Oct 30 22:33:48.718858 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.718748    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-istio-envoy\") pod \"cf853efa-c753-4ce6-834d-c315fba26177\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") "
Oct 30 22:33:48.718858 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.718843    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf853efa-c753-4ce6-834d-c315fba26177-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf853efa-c753-4ce6-834d-c315fba26177-istio-podinfo\") pod \"cf853efa-c753-4ce6-834d-c315fba26177\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") "
Oct 30 22:33:48.718964 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.718880    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf853efa-c753-4ce6-834d-c315fba26177-istio-token\" (UniqueName: \"kubernetes.io/projected/cf853efa-c753-4ce6-834d-c315fba26177-istio-token\") pod \"cf853efa-c753-4ce6-834d-c315fba26177\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") "
Oct 30 22:33:48.718964 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.718920    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-istio-data\") pod \"cf853efa-c753-4ce6-834d-c315fba26177\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") "
Oct 30 22:33:48.719041 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.718971    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-workload-socket" pod "cf853efa-c753-4ce6-834d-c315fba26177" (UID: "cf853efa-c753-4ce6-834d-c315fba26177"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.719041 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.719025    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-credential-socket\") pod \"cf853efa-c753-4ce6-834d-c315fba26177\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") "
Oct 30 22:33:48.719155 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.719059    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-workload-certs\") pod \"cf853efa-c753-4ce6-834d-c315fba26177\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") "
Oct 30 22:33:48.719314 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.719277    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-istio-data" pod "cf853efa-c753-4ce6-834d-c315fba26177" (UID: "cf853efa-c753-4ce6-834d-c315fba26177"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.719405 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.719302    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-credential-socket" pod "cf853efa-c753-4ce6-834d-c315fba26177" (UID: "cf853efa-c753-4ce6-834d-c315fba26177"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.719405 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.719384    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf853efa-c753-4ce6-834d-c315fba26177-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf853efa-c753-4ce6-834d-c315fba26177-istiod-ca-cert\") pod \"cf853efa-c753-4ce6-834d-c315fba26177\" (UID: \"cf853efa-c753-4ce6-834d-c315fba26177\") "
Oct 30 22:33:48.719576 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.719533    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-workload-certs" pod "cf853efa-c753-4ce6-834d-c315fba26177" (UID: "cf853efa-c753-4ce6-834d-c315fba26177"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.719853 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.719782    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf853efa-c753-4ce6-834d-c315fba26177-istiod-ca-cert" pod "cf853efa-c753-4ce6-834d-c315fba26177" (UID: "cf853efa-c753-4ce6-834d-c315fba26177"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:48.720373 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.720351    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.720441 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.720380    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29631b37-3676-4395-a55d-f09f9b3e0025-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.720441 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.720395    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.720441 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.720410    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.720441 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.720424    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.720441 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.720437    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.720681 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.720450    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.720681 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.720463    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhxbq\" (UniqueName: \"kubernetes.io/projected/35eb0630-0a80-41fc-ab11-6d2577213921-kube-api-access-mhxbq\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.720681 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.720476    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35eb0630-0a80-41fc-ab11-6d2577213921-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.720681 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.720489    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29631b37-3676-4395-a55d-f09f9b3e0025-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.720681 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.720502    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35eb0630-0a80-41fc-ab11-6d2577213921-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.720681 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.720514    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35eb0630-0a80-41fc-ab11-6d2577213921-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.720681 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.720528    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.720681 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.720540    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf853efa-c753-4ce6-834d-c315fba26177-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.720681 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.720565    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29631b37-3676-4395-a55d-f09f9b3e0025-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.720681 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.720578    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.720681 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.720592    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhplt\" (UniqueName: \"kubernetes.io/projected/29631b37-3676-4395-a55d-f09f9b3e0025-kube-api-access-lhplt\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.721623 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.721602    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf853efa-c753-4ce6-834d-c315fba26177-istio-token" pod "cf853efa-c753-4ce6-834d-c315fba26177" (UID: "cf853efa-c753-4ce6-834d-c315fba26177"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:48.721864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.721844    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-istio-envoy" pod "cf853efa-c753-4ce6-834d-c315fba26177" (UID: "cf853efa-c753-4ce6-834d-c315fba26177"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.721864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.721852    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf853efa-c753-4ce6-834d-c315fba26177-istio-podinfo" pod "cf853efa-c753-4ce6-834d-c315fba26177" (UID: "cf853efa-c753-4ce6-834d-c315fba26177"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:33:48.721959 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.721851    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf853efa-c753-4ce6-834d-c315fba26177-kube-api-access-nwxjq" pod "cf853efa-c753-4ce6-834d-c315fba26177" (UID: "cf853efa-c753-4ce6-834d-c315fba26177"). InnerVolumeSpecName "kube-api-access-nwxjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:48.724799 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.724771    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:48.822776 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.822741    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwxjq\" (UniqueName: \"kubernetes.io/projected/cf853efa-c753-4ce6-834d-c315fba26177-kube-api-access-nwxjq\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.822776 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.822777    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf853efa-c753-4ce6-834d-c315fba26177-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.823007 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.822791    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf853efa-c753-4ce6-834d-c315fba26177-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.823007 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.822803    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf853efa-c753-4ce6-834d-c315fba26177-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.823084 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.823058    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"]
Oct 30 22:33:48.835777 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:48.835739    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-w7p95"]
Oct 30 22:33:49.521091 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:49.521051    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"
Oct 30 22:33:49.521639 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:49.521045    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t" event={"ID":"cf853efa-c753-4ce6-834d-c315fba26177","Type":"ContainerDied","Data":"db20aa1315b1b25971caefb69637982b86cc3aede77b66de424387f55af1f25a"}
Oct 30 22:33:49.521639 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:49.521311    4889 scope.go:122] "RemoveContainer" containerID="0e6e14c1e558127bda6e7dde7960e9a93c3dab845c6761f24e46a63b131307ab"
Oct 30 22:33:49.523282 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:49.523260    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh" event={"ID":"29631b37-3676-4395-a55d-f09f9b3e0025","Type":"ContainerDied","Data":"5ce0e743d1f195655df4f44f95c29ebced991f3af4e110285a6531800456114f"}
Oct 30 22:33:49.523399 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:49.523316    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"
Oct 30 22:33:49.525276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:49.525254    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp" event={"ID":"35eb0630-0a80-41fc-ab11-6d2577213921","Type":"ContainerDied","Data":"f3dfa278436786e524557cd3fa05e99d7d6eaaec3c794c93de350468c664ee1b"}
Oct 30 22:33:49.525389 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:49.525301    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"
Oct 30 22:33:49.535164 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:49.535139    4889 scope.go:122] "RemoveContainer" containerID="81a7d52cdd909cd00bcba9e276b378ab0baeb3a27f535e841886401174e2459b"
Oct 30 22:33:49.550216 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:49.550189    4889 scope.go:122] "RemoveContainer" containerID="7a75e86e8983ed0303c752dc60bd99135a363b39c1bdccb7b47715d8702d5710"
Oct 30 22:33:49.586724 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:49.586692    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"]
Oct 30 22:33:49.598308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:49.598280    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-mhwhp"]
Oct 30 22:33:49.631350 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:49.631315    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"]
Oct 30 22:33:49.636168 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:49.636145    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-78w6t"]
Oct 30 22:33:49.666942 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:49.666911    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"]
Oct 30 22:33:49.671786 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:49.671759    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-7v8wh"]
Oct 30 22:33:50.106026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:50.105993    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:50.777016 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:50.776983    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03297afe-0b96-48ea-bcb8-849f2cd95bb1" path="/var/lib/kubelet/pods/03297afe-0b96-48ea-bcb8-849f2cd95bb1/volumes"
Oct 30 22:33:50.777557 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:50.777543    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29631b37-3676-4395-a55d-f09f9b3e0025" path="/var/lib/kubelet/pods/29631b37-3676-4395-a55d-f09f9b3e0025/volumes"
Oct 30 22:33:50.777994 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:50.777981    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35eb0630-0a80-41fc-ab11-6d2577213921" path="/var/lib/kubelet/pods/35eb0630-0a80-41fc-ab11-6d2577213921/volumes"
Oct 30 22:33:50.778788 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:50.778773    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf853efa-c753-4ce6-834d-c315fba26177" path="/var/lib/kubelet/pods/cf853efa-c753-4ce6-834d-c315fba26177/volumes"
Oct 30 22:33:51.454962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:51.454925    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft"]
Oct 30 22:33:51.455235 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:51.455195    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft" podUID="09676531-ce1f-4950-b903-730935715f8a" containerName="maas-api" containerID="cri-o://9cf2b97303353fbcec60dd07aa09c4ee696518ad02c7b39256cf49d7dbeac310" gracePeriod=30
Oct 30 22:33:51.603760 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:51.603727    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:51.778862 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:51.778835    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft"
Oct 30 22:33:51.782919 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:51.782888    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:51.884246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:51.879836    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09676531-ce1f-4950-b903-730935715f8a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09676531-ce1f-4950-b903-730935715f8a-maas-api-tls\") pod \"09676531-ce1f-4950-b903-730935715f8a\" (UID: \"09676531-ce1f-4950-b903-730935715f8a\") "
Oct 30 22:33:51.884246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:51.880056    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09676531-ce1f-4950-b903-730935715f8a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09676531-ce1f-4950-b903-730935715f8a-openshift-service-ca\") pod \"09676531-ce1f-4950-b903-730935715f8a\" (UID: \"09676531-ce1f-4950-b903-730935715f8a\") "
Oct 30 22:33:51.884246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:51.880117    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09676531-ce1f-4950-b903-730935715f8a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/09676531-ce1f-4950-b903-730935715f8a-metrics-certs\") pod \"09676531-ce1f-4950-b903-730935715f8a\" (UID: \"09676531-ce1f-4950-b903-730935715f8a\") "
Oct 30 22:33:51.884246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:51.880364    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09676531-ce1f-4950-b903-730935715f8a-kube-api-access-569ql\" (UniqueName: \"kubernetes.io/projected/09676531-ce1f-4950-b903-730935715f8a-kube-api-access-569ql\") pod \"09676531-ce1f-4950-b903-730935715f8a\" (UID: \"09676531-ce1f-4950-b903-730935715f8a\") "
Oct 30 22:33:51.932285 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:51.886680    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09676531-ce1f-4950-b903-730935715f8a-openshift-service-ca" pod "09676531-ce1f-4950-b903-730935715f8a" (UID: "09676531-ce1f-4950-b903-730935715f8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:52.024309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:52.024141    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:52.050633 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:52.050545    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09676531-ce1f-4950-b903-730935715f8a-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:52.170799 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:52.169657    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09676531-ce1f-4950-b903-730935715f8a-metrics-certs" pod "09676531-ce1f-4950-b903-730935715f8a" (UID: "09676531-ce1f-4950-b903-730935715f8a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:52.190622 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:52.190582    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:52.196250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:52.192913    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09676531-ce1f-4950-b903-730935715f8a-kube-api-access-569ql" pod "09676531-ce1f-4950-b903-730935715f8a" (UID: "09676531-ce1f-4950-b903-730935715f8a"). InnerVolumeSpecName "kube-api-access-569ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:52.204015 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:52.203974    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09676531-ce1f-4950-b903-730935715f8a-maas-api-tls" pod "09676531-ce1f-4950-b903-730935715f8a" (UID: "09676531-ce1f-4950-b903-730935715f8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:52.261153 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:52.261116    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-569ql\" (UniqueName: \"kubernetes.io/projected/09676531-ce1f-4950-b903-730935715f8a-kube-api-access-569ql\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:52.261153 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:52.261151    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09676531-ce1f-4950-b903-730935715f8a-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:52.261153 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:52.261160    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09676531-ce1f-4950-b903-730935715f8a-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:52.401299 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:52.400610    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:52.654172 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:52.642687    4889 generic.go:356] "Generic (PLEG): container finished" podID="09676531-ce1f-4950-b903-730935715f8a" containerID="9cf2b97303353fbcec60dd07aa09c4ee696518ad02c7b39256cf49d7dbeac310" exitCode=0
Oct 30 22:33:52.654172 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:52.642736    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft" event={"ID":"09676531-ce1f-4950-b903-730935715f8a","Type":"ContainerDied","Data":"9cf2b97303353fbcec60dd07aa09c4ee696518ad02c7b39256cf49d7dbeac310"}
Oct 30 22:33:52.654172 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:52.642788    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft" event={"ID":"09676531-ce1f-4950-b903-730935715f8a","Type":"ContainerDied","Data":"a02cfc44b6d11cc2fd4f09af2c4b5fc021809ffa450acac93a246e17807c4e66"}
Oct 30 22:33:52.654172 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:52.642816    4889 scope.go:122] "RemoveContainer" containerID="9cf2b97303353fbcec60dd07aa09c4ee696518ad02c7b39256cf49d7dbeac310"
Oct 30 22:33:52.654172 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:52.643176    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft"
Oct 30 22:33:52.697259 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:52.696604    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:52.756322 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:52.753803    4889 scope.go:122] "RemoveContainer" containerID="9cf2b97303353fbcec60dd07aa09c4ee696518ad02c7b39256cf49d7dbeac310"
Oct 30 22:33:52.756322 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:33:52.754276    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cf2b97303353fbcec60dd07aa09c4ee696518ad02c7b39256cf49d7dbeac310\": container with ID starting with 9cf2b97303353fbcec60dd07aa09c4ee696518ad02c7b39256cf49d7dbeac310 not found: ID does not exist" containerID="9cf2b97303353fbcec60dd07aa09c4ee696518ad02c7b39256cf49d7dbeac310"
Oct 30 22:33:52.756322 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:52.754315    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cf2b97303353fbcec60dd07aa09c4ee696518ad02c7b39256cf49d7dbeac310"} err="failed to get container status \"9cf2b97303353fbcec60dd07aa09c4ee696518ad02c7b39256cf49d7dbeac310\": rpc error: code = NotFound desc = could not find container \"9cf2b97303353fbcec60dd07aa09c4ee696518ad02c7b39256cf49d7dbeac310\": container with ID starting with 9cf2b97303353fbcec60dd07aa09c4ee696518ad02c7b39256cf49d7dbeac310 not found: ID does not exist"
Oct 30 22:33:52.826148 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:52.826107    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft"]
Oct 30 22:33:52.826148 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:52.826155    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f443d6-7c96986b84-52sft"]
Oct 30 22:33:52.864207 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:52.864174    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:54.083207 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:54.083168    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq"]
Oct 30 22:33:54.117115 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:54.117082    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6"]
Oct 30 22:33:54.288352 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:54.288315    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:54.713202 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:54.713164    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:54.933263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:54.930611    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09676531-ce1f-4950-b903-730935715f8a" path="/var/lib/kubelet/pods/09676531-ce1f-4950-b903-730935715f8a/volumes"
Oct 30 22:33:54.933263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:54.931822    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:56.642593 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:56.642552    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:58.367501 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:58.367462    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:59.084696 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.084653    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq" podUID="6d3de30f-5af9-4195-bb1b-1de3f0643e06" containerName="payload-processing" containerID="cri-o://55f97ddae8ee5f2ee9ff2a61403960ecd5e006882fbe61d12f5f20834e417d29" gracePeriod=25
Oct 30 22:33:59.118057 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.118018    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6" podUID="40454feb-b149-4312-a275-62e8dbac9c23" containerName="payload-pre-processing" containerID="cri-o://1f6bd96dc1f4358e233171409c4501453ba23025397870c3bec30fae9b1571a1" gracePeriod=25
Oct 30 22:33:59.430175 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.430143    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq"
Oct 30 22:33:59.474098 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.474068    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6"
Oct 30 22:33:59.480774 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.480743    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d3de30f-5af9-4195-bb1b-1de3f0643e06-kube-api-access-v74ld\" (UniqueName: \"kubernetes.io/projected/6d3de30f-5af9-4195-bb1b-1de3f0643e06-kube-api-access-v74ld\") pod \"6d3de30f-5af9-4195-bb1b-1de3f0643e06\" (UID: \"6d3de30f-5af9-4195-bb1b-1de3f0643e06\") "
Oct 30 22:33:59.480953 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.480934    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d3de30f-5af9-4195-bb1b-1de3f0643e06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d3de30f-5af9-4195-bb1b-1de3f0643e06-plugins-config-volume\") pod \"6d3de30f-5af9-4195-bb1b-1de3f0643e06\" (UID: \"6d3de30f-5af9-4195-bb1b-1de3f0643e06\") "
Oct 30 22:33:59.481352 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.481326    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d3de30f-5af9-4195-bb1b-1de3f0643e06-plugins-config-volume" pod "6d3de30f-5af9-4195-bb1b-1de3f0643e06" (UID: "6d3de30f-5af9-4195-bb1b-1de3f0643e06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:59.482060 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.482038    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d3de30f-5af9-4195-bb1b-1de3f0643e06-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:59.483479 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.483455    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d3de30f-5af9-4195-bb1b-1de3f0643e06-kube-api-access-v74ld" pod "6d3de30f-5af9-4195-bb1b-1de3f0643e06" (UID: "6d3de30f-5af9-4195-bb1b-1de3f0643e06"). InnerVolumeSpecName "kube-api-access-v74ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:59.582551 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.582514    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40454feb-b149-4312-a275-62e8dbac9c23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40454feb-b149-4312-a275-62e8dbac9c23-plugins-config-volume\") pod \"40454feb-b149-4312-a275-62e8dbac9c23\" (UID: \"40454feb-b149-4312-a275-62e8dbac9c23\") "
Oct 30 22:33:59.582749 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.582624    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40454feb-b149-4312-a275-62e8dbac9c23-kube-api-access-cfkts\" (UniqueName: \"kubernetes.io/projected/40454feb-b149-4312-a275-62e8dbac9c23-kube-api-access-cfkts\") pod \"40454feb-b149-4312-a275-62e8dbac9c23\" (UID: \"40454feb-b149-4312-a275-62e8dbac9c23\") "
Oct 30 22:33:59.582899 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.582870    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40454feb-b149-4312-a275-62e8dbac9c23-plugins-config-volume" pod "40454feb-b149-4312-a275-62e8dbac9c23" (UID: "40454feb-b149-4312-a275-62e8dbac9c23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:59.583565 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.583542    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v74ld\" (UniqueName: \"kubernetes.io/projected/6d3de30f-5af9-4195-bb1b-1de3f0643e06-kube-api-access-v74ld\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:59.583655 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.583566    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40454feb-b149-4312-a275-62e8dbac9c23-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:59.585055 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.585031    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40454feb-b149-4312-a275-62e8dbac9c23-kube-api-access-cfkts" pod "40454feb-b149-4312-a275-62e8dbac9c23" (UID: "40454feb-b149-4312-a275-62e8dbac9c23"). InnerVolumeSpecName "kube-api-access-cfkts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:59.685188 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.685100    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfkts\" (UniqueName: \"kubernetes.io/projected/40454feb-b149-4312-a275-62e8dbac9c23-kube-api-access-cfkts\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:33:59.721577 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.721539    4889 generic.go:356] "Generic (PLEG): container finished" podID="40454feb-b149-4312-a275-62e8dbac9c23" containerID="1f6bd96dc1f4358e233171409c4501453ba23025397870c3bec30fae9b1571a1" exitCode=0
Oct 30 22:33:59.721777 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.721585    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6"
Oct 30 22:33:59.721777 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.721621    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6" event={"ID":"40454feb-b149-4312-a275-62e8dbac9c23","Type":"ContainerDied","Data":"1f6bd96dc1f4358e233171409c4501453ba23025397870c3bec30fae9b1571a1"}
Oct 30 22:33:59.721777 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.721675    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6" event={"ID":"40454feb-b149-4312-a275-62e8dbac9c23","Type":"ContainerDied","Data":"2da3336ddc6c293ec4eab9f623a2c10afb008bdce78f4b190a97445e29a9dbf4"}
Oct 30 22:33:59.721777 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.721701    4889 scope.go:122] "RemoveContainer" containerID="1f6bd96dc1f4358e233171409c4501453ba23025397870c3bec30fae9b1571a1"
Oct 30 22:33:59.723539 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.723500    4889 generic.go:356] "Generic (PLEG): container finished" podID="6d3de30f-5af9-4195-bb1b-1de3f0643e06" containerID="55f97ddae8ee5f2ee9ff2a61403960ecd5e006882fbe61d12f5f20834e417d29" exitCode=0
Oct 30 22:33:59.723667 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.723544    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq"
Oct 30 22:33:59.723667 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.723609    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq" event={"ID":"6d3de30f-5af9-4195-bb1b-1de3f0643e06","Type":"ContainerDied","Data":"55f97ddae8ee5f2ee9ff2a61403960ecd5e006882fbe61d12f5f20834e417d29"}
Oct 30 22:33:59.723667 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.723648    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq" event={"ID":"6d3de30f-5af9-4195-bb1b-1de3f0643e06","Type":"ContainerDied","Data":"e3e09aa9e1ab5168a9416d506ac3abaaabea7ae1b1537e2b2cdaacb33ca10ad8"}
Oct 30 22:33:59.734535 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.734458    4889 scope.go:122] "RemoveContainer" containerID="1f6bd96dc1f4358e233171409c4501453ba23025397870c3bec30fae9b1571a1"
Oct 30 22:33:59.734814 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:33:59.734791    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f6bd96dc1f4358e233171409c4501453ba23025397870c3bec30fae9b1571a1\": container with ID starting with 1f6bd96dc1f4358e233171409c4501453ba23025397870c3bec30fae9b1571a1 not found: ID does not exist" containerID="1f6bd96dc1f4358e233171409c4501453ba23025397870c3bec30fae9b1571a1"
Oct 30 22:33:59.734924 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.734823    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f6bd96dc1f4358e233171409c4501453ba23025397870c3bec30fae9b1571a1"} err="failed to get container status \"1f6bd96dc1f4358e233171409c4501453ba23025397870c3bec30fae9b1571a1\": rpc error: code = NotFound desc = could not find container \"1f6bd96dc1f4358e233171409c4501453ba23025397870c3bec30fae9b1571a1\": container with ID starting with 1f6bd96dc1f4358e233171409c4501453ba23025397870c3bec30fae9b1571a1 not found: ID does not exist"
Oct 30 22:33:59.734924 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.734845    4889 scope.go:122] "RemoveContainer" containerID="55f97ddae8ee5f2ee9ff2a61403960ecd5e006882fbe61d12f5f20834e417d29"
Oct 30 22:33:59.751521 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.751497    4889 scope.go:122] "RemoveContainer" containerID="55f97ddae8ee5f2ee9ff2a61403960ecd5e006882fbe61d12f5f20834e417d29"
Oct 30 22:33:59.751849 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:33:59.751831    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55f97ddae8ee5f2ee9ff2a61403960ecd5e006882fbe61d12f5f20834e417d29\": container with ID starting with 55f97ddae8ee5f2ee9ff2a61403960ecd5e006882fbe61d12f5f20834e417d29 not found: ID does not exist" containerID="55f97ddae8ee5f2ee9ff2a61403960ecd5e006882fbe61d12f5f20834e417d29"
Oct 30 22:33:59.751902 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.751856    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55f97ddae8ee5f2ee9ff2a61403960ecd5e006882fbe61d12f5f20834e417d29"} err="failed to get container status \"55f97ddae8ee5f2ee9ff2a61403960ecd5e006882fbe61d12f5f20834e417d29\": rpc error: code = NotFound desc = could not find container \"55f97ddae8ee5f2ee9ff2a61403960ecd5e006882fbe61d12f5f20834e417d29\": container with ID starting with 55f97ddae8ee5f2ee9ff2a61403960ecd5e006882fbe61d12f5f20834e417d29 not found: ID does not exist"
Oct 30 22:33:59.772353 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.772322    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6"]
Oct 30 22:33:59.798261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.785455    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bf9c2f-69f6d78d99-b2xh6"]
Oct 30 22:33:59.815973 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.815202    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:59.846727 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.846688    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq"]
Oct 30 22:33:59.858145 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.855420    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bf9c2f-59899c7df-58lpq"]
Oct 30 22:33:59.980751 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:33:59.980641    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:00.207248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:00.202789    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:00.777532 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:00.777499    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40454feb-b149-4312-a275-62e8dbac9c23" path="/var/lib/kubelet/pods/40454feb-b149-4312-a275-62e8dbac9c23/volumes"
Oct 30 22:34:00.778044 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:00.778029    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d3de30f-5af9-4195-bb1b-1de3f0643e06" path="/var/lib/kubelet/pods/6d3de30f-5af9-4195-bb1b-1de3f0643e06/volumes"
Oct 30 22:34:01.670638 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:01.670604    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:01.857133 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:01.857095    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:02.030251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:02.029386    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:04.173273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:04.166013    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:05.933099 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:05.933065    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:06.716253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:06.708037    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:07.240266 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:07.240200    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:07.271739 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:07.271699    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6" podUID="437136a8-5481-471f-9ec3-ae970c671a60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded"
Oct 30 22:34:07.744248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:07.743771    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:08.898016 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:08.897976    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:09.086421 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:09.086383    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:09.289709 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:09.289673    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:10.306797 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:34:10.306758    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:34:11.663110 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:11.663076    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:12.460642 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:12.460602    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:12.645407 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:12.645373    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:13.569061 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:13.569022    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:14.780947 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.780916    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"]
Oct 30 22:34:14.781692 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.781674    4889 cpu_manager.go:414] "containerMap: removing container" podUID="cf853efa-c753-4ce6-834d-c315fba26177" containerName="istio-proxy"
Oct 30 22:34:14.781692 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.781692    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf853efa-c753-4ce6-834d-c315fba26177" containerName="istio-proxy"
Oct 30 22:34:14.781787 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.781711    4889 cpu_manager.go:414] "containerMap: removing container" podUID="40454feb-b149-4312-a275-62e8dbac9c23" containerName="payload-pre-processing"
Oct 30 22:34:14.781787 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.781717    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40454feb-b149-4312-a275-62e8dbac9c23" containerName="payload-pre-processing"
Oct 30 22:34:14.781787 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.781758    4889 cpu_manager.go:414] "containerMap: removing container" podUID="29631b37-3676-4395-a55d-f09f9b3e0025" containerName="istio-proxy"
Oct 30 22:34:14.781787 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.781764    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29631b37-3676-4395-a55d-f09f9b3e0025" containerName="istio-proxy"
Oct 30 22:34:14.781787 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.781775    4889 cpu_manager.go:414] "containerMap: removing container" podUID="03297afe-0b96-48ea-bcb8-849f2cd95bb1" containerName="istio-proxy"
Oct 30 22:34:14.781787 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.781779    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03297afe-0b96-48ea-bcb8-849f2cd95bb1" containerName="istio-proxy"
Oct 30 22:34:14.781981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.781794    4889 cpu_manager.go:414] "containerMap: removing container" podUID="6d3de30f-5af9-4195-bb1b-1de3f0643e06" containerName="payload-processing"
Oct 30 22:34:14.781981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.781799    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d3de30f-5af9-4195-bb1b-1de3f0643e06" containerName="payload-processing"
Oct 30 22:34:14.781981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.781809    4889 cpu_manager.go:414] "containerMap: removing container" podUID="09676531-ce1f-4950-b903-730935715f8a" containerName="maas-api"
Oct 30 22:34:14.781981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.781813    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09676531-ce1f-4950-b903-730935715f8a" containerName="maas-api"
Oct 30 22:34:14.781981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.781820    4889 cpu_manager.go:414] "containerMap: removing container" podUID="cf853efa-c753-4ce6-834d-c315fba26177" containerName="istio-proxy"
Oct 30 22:34:14.781981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.781824    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf853efa-c753-4ce6-834d-c315fba26177" containerName="istio-proxy"
Oct 30 22:34:14.781981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.781830    4889 cpu_manager.go:414] "containerMap: removing container" podUID="35eb0630-0a80-41fc-ab11-6d2577213921" containerName="istio-proxy"
Oct 30 22:34:14.781981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.781835    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35eb0630-0a80-41fc-ab11-6d2577213921" containerName="istio-proxy"
Oct 30 22:34:14.781981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.781962    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="29631b37-3676-4395-a55d-f09f9b3e0025" containerName="istio-proxy"
Oct 30 22:34:14.781981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.781972    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="03297afe-0b96-48ea-bcb8-849f2cd95bb1" containerName="istio-proxy"
Oct 30 22:34:14.781981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.781979    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d3de30f-5af9-4195-bb1b-1de3f0643e06" containerName="payload-processing"
Oct 30 22:34:14.781981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.781986    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="09676531-ce1f-4950-b903-730935715f8a" containerName="maas-api"
Oct 30 22:34:14.782482 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.781994    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf853efa-c753-4ce6-834d-c315fba26177" containerName="istio-proxy"
Oct 30 22:34:14.782482 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.782000    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="35eb0630-0a80-41fc-ab11-6d2577213921" containerName="istio-proxy"
Oct 30 22:34:14.782482 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.782007    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf853efa-c753-4ce6-834d-c315fba26177" containerName="istio-proxy"
Oct 30 22:34:14.782482 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.782014    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="40454feb-b149-4312-a275-62e8dbac9c23" containerName="payload-pre-processing"
Oct 30 22:34:14.785789 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.785765    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:14.805678 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.805640    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"]
Oct 30 22:34:14.891984 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.891940    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-podinfo\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:14.892192 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.892074    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-workload-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:14.892192 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.892185    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-credential-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:14.892342 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.892248    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z6zl\" (UniqueName: \"kubernetes.io/projected/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-kube-api-access-4z6zl\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:14.892454 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.892429    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-envoy\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:14.892551 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.892502    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-data\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:14.892835 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.892812    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-token\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:14.892911 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.892857    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-workload-certs\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:14.892975 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.892958    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istiod-ca-cert\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:14.994712 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.994678    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-token\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:14.994712 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.994718    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-workload-certs\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:14.994987 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.994742    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istiod-ca-cert\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:14.994987 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.994790    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-podinfo\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:14.994987 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.994830    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-workload-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:14.994987 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.994894    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-credential-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:14.994987 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.994944    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z6zl\" (UniqueName: \"kubernetes.io/projected/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-kube-api-access-4z6zl\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:14.995318 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.995094    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-workload-certs\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:14.995388 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.995348    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-workload-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:14.995609 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.995472    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-envoy\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:14.995609 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.995532    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-credential-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:14.995609 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.995537    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-data\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:14.996104 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.995990    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istiod-ca-cert\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:14.996214 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.996101    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-data\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:14.998393 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.998369    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-podinfo\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:14.998510 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:14.998393    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-envoy\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:15.013856 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:15.013819    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z6zl\" (UniqueName: \"kubernetes.io/projected/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-kube-api-access-4z6zl\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:15.014687 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:15.014658    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-token\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:15.118323 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:15.118240    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:15.369814 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:15.369474    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:15.460092 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:15.460062    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"]
Oct 30 22:34:15.460278 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:34:15.460255    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb211a3e6_ec7c_418c_9fa1_a2546cacd7f7.slice/crio-75e589eba7ca7acfae7e2540f468ae578d3e3d607275f3050dd49c83345fd43d WatchSource:0}: Error finding container 75e589eba7ca7acfae7e2540f468ae578d3e3d607275f3050dd49c83345fd43d: Status 404 returned error can't find the container with id 75e589eba7ca7acfae7e2540f468ae578d3e3d607275f3050dd49c83345fd43d
Oct 30 22:34:15.463615 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:15.463581    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:34:15.463703 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:15.463650    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:34:15.463703 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:15.463681    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:34:15.886824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:15.886785    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:16.286570 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.286528    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6" event={"ID":"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7","Type":"ContainerStarted","Data":"1f1b7d34f491636176e9f6fcd1d55b356b2b4a8da3a55c76055e35d1daa198a8"}
Oct 30 22:34:16.286766 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.286597    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6" event={"ID":"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7","Type":"ContainerStarted","Data":"75e589eba7ca7acfae7e2540f468ae578d3e3d607275f3050dd49c83345fd43d"}
Oct 30 22:34:16.321820 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.321771    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:16.672980 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.672875    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6" podStartSLOduration=2.672855883 podStartE2EDuration="2.672855883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:16.341031504 +0000 UTC m=+1641.697397818" watchObservedRunningTime="2026-10-30 22:34:16.672855883 +0000 UTC m=+1642.029222200"
Oct 30 22:34:16.681515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.681479    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"]
Oct 30 22:34:16.686909 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.686728    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:16.730254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.716048    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f9d87b-openshift-default-dockercfg-b6gdf\""
Oct 30 22:34:16.750292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.746245    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istiod-ca-cert\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:16.750292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.747540    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-credential-socket\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:16.750292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.747582    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-workload-certs\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:16.750292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.747666    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-data\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:16.750292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.747690    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-podinfo\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:16.750292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.747721    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-envoy\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:16.750292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.747771    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-workload-socket\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:16.750292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.747839    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-token\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:16.750292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.747869    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc6lp\" (UniqueName: \"kubernetes.io/projected/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-kube-api-access-lc6lp\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:16.753507 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.753475    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"]
Oct 30 22:34:16.853251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.850846    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-workload-certs\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:16.853251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.850971    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-data\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:16.853251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.851003    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-podinfo\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:16.853251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.851035    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-envoy\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:16.853251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.851093    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-workload-socket\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:16.853251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.851171    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-token\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:16.853251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.851205    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc6lp\" (UniqueName: \"kubernetes.io/projected/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-kube-api-access-lc6lp\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:16.853251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.851429    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istiod-ca-cert\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:16.853251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.851572    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-credential-socket\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:16.853251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.852033    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-credential-socket\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:16.897252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.892622    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:16.907248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:16.904482    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-workload-certs\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:17.085326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:17.082630    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-podinfo\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:17.085326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:17.082817    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-workload-socket\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:17.085326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:17.083273    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istiod-ca-cert\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:17.085326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:17.083376    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-data\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:17.085326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:17.083963    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc6lp\" (UniqueName: \"kubernetes.io/projected/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-kube-api-access-lc6lp\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:17.085326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:17.084244    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-envoy\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:17.104287 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:17.086555    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-token\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:17.120507 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:17.120477    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:17.122199 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:17.122169    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 30 22:34:17.122455 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:17.122390    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6" podUID="b211a3e6-ec7c-418c-9fa1-a2546cacd7f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 30 22:34:17.360506 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:17.360428    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:17.589253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:17.580785    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:17.793139 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:17.779417    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:17.801254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:17.799297    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"]
Oct 30 22:34:17.816518 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:17.806740    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg"]
Oct 30 22:34:17.853682 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:34:17.853646    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b49c0bf_aa2d_4f17_8a09_831d9b8bc459.slice/crio-cc8d5a3d4b48d633f0f6f2a3143bddea1dc9cc43b545f65cc3a5f355a63c03bf WatchSource:0}: Error finding container cc8d5a3d4b48d633f0f6f2a3143bddea1dc9cc43b545f65cc3a5f355a63c03bf: Status 404 returned error can't find the container with id cc8d5a3d4b48d633f0f6f2a3143bddea1dc9cc43b545f65cc3a5f355a63c03bf
Oct 30 22:34:17.859640 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:17.858276    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w"]
Oct 30 22:34:17.859640 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:17.858519    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:34:17.859640 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:17.858593    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:34:17.859640 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:17.858632    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:34:18.105701 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:18.051562    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:18.158840 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:18.118726    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 30 22:34:18.158840 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:18.118787    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6" podUID="b211a3e6-ec7c-418c-9fa1-a2546cacd7f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 30 22:34:18.224273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:18.223755    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:18.355756 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:18.355622    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk" event={"ID":"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459","Type":"ContainerStarted","Data":"bf0cb469820dc9eb0b5a8ed43d63125eaa4f4e1a0bd2199e54f55889227bc390"}
Oct 30 22:34:18.355756 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:18.355678    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk" event={"ID":"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459","Type":"ContainerStarted","Data":"cc8d5a3d4b48d633f0f6f2a3143bddea1dc9cc43b545f65cc3a5f355a63c03bf"}
Oct 30 22:34:18.361050 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:18.361019    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:18.382725 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:18.382681    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": read tcp 10.132.0.2:50754->10.132.0.153:15021: read: connection reset by peer" start-of-body=
Oct 30 22:34:18.382884 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:18.382748    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z" podUID="8b34996e-28a0-4cbf-b0fa-4615ad9fccaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": read tcp 10.132.0.2:50754->10.132.0.153:15021: read: connection reset by peer"
Oct 30 22:34:19.132250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:19.131663    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 30 22:34:19.132250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:19.131728    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6" podUID="b211a3e6-ec7c-418c-9fa1-a2546cacd7f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 30 22:34:19.370593 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:19.370568    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:19.371581 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:19.371554    4889 generic.go:356] "Generic (PLEG): container finished" podID="8b34996e-28a0-4cbf-b0fa-4615ad9fccaa" containerID="13a8216f5efc056073c38bf635471fcaae2938ab56d8d1df481b9d5dc2188a75" exitCode=0
Oct 30 22:34:19.372882 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:19.372857    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z" event={"ID":"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa","Type":"ContainerDied","Data":"13a8216f5efc056073c38bf635471fcaae2938ab56d8d1df481b9d5dc2188a75"}
Oct 30 22:34:19.373095 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:19.373075    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:19.373779 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:19.373744    4889 scope.go:122] "RemoveContainer" containerID="13a8216f5efc056073c38bf635471fcaae2938ab56d8d1df481b9d5dc2188a75"
Oct 30 22:34:19.374492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:19.374468    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:34:19.375938 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:19.375910    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:34:19.376011 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:19.375981    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:34:19.376050 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:19.376018    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:34:19.442939 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:19.441907    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk" podStartSLOduration=3.441888561 podStartE2EDuration="3.441888561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:18.399259391 +0000 UTC m=+1643.755625708" watchObservedRunningTime="2026-10-30 22:34:19.441888561 +0000 UTC m=+1644.798254876"
Oct 30 22:34:19.683675 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:19.677557    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:20.135310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:20.121679    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 30 22:34:20.135310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:20.121742    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6" podUID="b211a3e6-ec7c-418c-9fa1-a2546cacd7f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 30 22:34:20.382893 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:20.382858    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z" event={"ID":"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa","Type":"ContainerStarted","Data":"02b2df1c5dfc1189bea3c629356c1ceae2b6ae0064ac550a33166e01262c0764"}
Oct 30 22:34:20.984070 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:20.984030    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:21.100283 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:21.100239    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:34:21.100471 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:21.100435    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 22:34:21.100517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:21.100480    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z" podUID="8b34996e-28a0-4cbf-b0fa-4615ad9fccaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 22:34:21.118707 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:21.118671    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 30 22:34:21.118996 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:21.118969    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6" podUID="b211a3e6-ec7c-418c-9fa1-a2546cacd7f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 30 22:34:21.195015 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:21.194979    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:22.098277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:22.098244    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 22:34:22.098468 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:22.098298    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z" podUID="8b34996e-28a0-4cbf-b0fa-4615ad9fccaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 22:34:22.119673 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:22.119638    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 30 22:34:22.120009 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:22.119969    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6" podUID="b211a3e6-ec7c-418c-9fa1-a2546cacd7f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 30 22:34:22.232613 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:22.232577    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:22.807841 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:22.807797    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg" podUID="5c0067cb-bba0-4ae1-9e07-3412034dac9b" containerName="payload-processing" containerID="cri-o://bc2b975ea14f5f2dc9df853978508c1f9fac9aa3b9a8e20d38d7546f0b1385a5" gracePeriod=25
Oct 30 22:34:22.859002 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:22.858965    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w" podUID="03c4d421-e943-45a1-91f9-71b9ec78daba" containerName="payload-pre-processing" containerID="cri-o://b8fef9f31a03651c323c69663c5892737e15772981e62c33e0369b589abe9c00" gracePeriod=25
Oct 30 22:34:23.103558 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.103096    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 22:34:23.103558 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.103161    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z" podUID="8b34996e-28a0-4cbf-b0fa-4615ad9fccaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 22:34:23.139464 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.139419    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:23.142595 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.142566    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:23.143483 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.143462    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:23.145311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.145286    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:34:23.322249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.306982    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5"]
Oct 30 22:34:23.342308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.328016    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:23.349339 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.348698    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5"
Oct 30 22:34:23.352194 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.352159    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5"]
Oct 30 22:34:23.361865 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.361837    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg"
Oct 30 22:34:23.363156 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.363135    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f9d87b-dockercfg-l8nmq\""
Oct 30 22:34:23.363443 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.363425    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f9d87b\""
Oct 30 22:34:23.367834 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.367807    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w"
Oct 30 22:34:23.439252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.434967    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03c4d421-e943-45a1-91f9-71b9ec78daba-kube-api-access-8qwlc\" (UniqueName: \"kubernetes.io/projected/03c4d421-e943-45a1-91f9-71b9ec78daba-kube-api-access-8qwlc\") pod \"03c4d421-e943-45a1-91f9-71b9ec78daba\" (UID: \"03c4d421-e943-45a1-91f9-71b9ec78daba\") "
Oct 30 22:34:23.439252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.435052    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c0067cb-bba0-4ae1-9e07-3412034dac9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0067cb-bba0-4ae1-9e07-3412034dac9b-plugins-config-volume\") pod \"5c0067cb-bba0-4ae1-9e07-3412034dac9b\" (UID: \"5c0067cb-bba0-4ae1-9e07-3412034dac9b\") "
Oct 30 22:34:23.472246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.462112    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c0067cb-bba0-4ae1-9e07-3412034dac9b-kube-api-access-zbfm2\" (UniqueName: \"kubernetes.io/projected/5c0067cb-bba0-4ae1-9e07-3412034dac9b-kube-api-access-zbfm2\") pod \"5c0067cb-bba0-4ae1-9e07-3412034dac9b\" (UID: \"5c0067cb-bba0-4ae1-9e07-3412034dac9b\") "
Oct 30 22:34:23.472246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.462296    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03c4d421-e943-45a1-91f9-71b9ec78daba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03c4d421-e943-45a1-91f9-71b9ec78daba-plugins-config-volume\") pod \"03c4d421-e943-45a1-91f9-71b9ec78daba\" (UID: \"03c4d421-e943-45a1-91f9-71b9ec78daba\") "
Oct 30 22:34:23.508481 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.500899    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74bb46da-f26a-4f88-85e2-c6dc0c305a7a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5\" (UID: \"74bb46da-f26a-4f88-85e2-c6dc0c305a7a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5"
Oct 30 22:34:23.508481 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.501090    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlx4b\" (UniqueName: \"kubernetes.io/projected/74bb46da-f26a-4f88-85e2-c6dc0c305a7a-kube-api-access-tlx4b\") pod \"payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5\" (UID: \"74bb46da-f26a-4f88-85e2-c6dc0c305a7a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5"
Oct 30 22:34:23.510392 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.508973    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03c4d421-e943-45a1-91f9-71b9ec78daba-plugins-config-volume" pod "03c4d421-e943-45a1-91f9-71b9ec78daba" (UID: "03c4d421-e943-45a1-91f9-71b9ec78daba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:23.538250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.535098    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c0067cb-bba0-4ae1-9e07-3412034dac9b-plugins-config-volume" pod "5c0067cb-bba0-4ae1-9e07-3412034dac9b" (UID: "5c0067cb-bba0-4ae1-9e07-3412034dac9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:23.568251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.540553    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c0067cb-bba0-4ae1-9e07-3412034dac9b-kube-api-access-zbfm2" pod "5c0067cb-bba0-4ae1-9e07-3412034dac9b" (UID: "5c0067cb-bba0-4ae1-9e07-3412034dac9b"). InnerVolumeSpecName "kube-api-access-zbfm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:23.568251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.540661    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c4d421-e943-45a1-91f9-71b9ec78daba-kube-api-access-8qwlc" pod "03c4d421-e943-45a1-91f9-71b9ec78daba" (UID: "03c4d421-e943-45a1-91f9-71b9ec78daba"). InnerVolumeSpecName "kube-api-access-8qwlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:23.608397 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.604415    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74bb46da-f26a-4f88-85e2-c6dc0c305a7a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5\" (UID: \"74bb46da-f26a-4f88-85e2-c6dc0c305a7a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5"
Oct 30 22:34:23.608397 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.604558    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlx4b\" (UniqueName: \"kubernetes.io/projected/74bb46da-f26a-4f88-85e2-c6dc0c305a7a-kube-api-access-tlx4b\") pod \"payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5\" (UID: \"74bb46da-f26a-4f88-85e2-c6dc0c305a7a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5"
Oct 30 22:34:23.608397 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.604796    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03c4d421-e943-45a1-91f9-71b9ec78daba-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:34:23.608397 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.604813    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qwlc\" (UniqueName: \"kubernetes.io/projected/03c4d421-e943-45a1-91f9-71b9ec78daba-kube-api-access-8qwlc\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:34:23.608397 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.604826    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0067cb-bba0-4ae1-9e07-3412034dac9b-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:34:23.608397 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.604839    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbfm2\" (UniqueName: \"kubernetes.io/projected/5c0067cb-bba0-4ae1-9e07-3412034dac9b-kube-api-access-zbfm2\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:34:23.629473 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.623829    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74bb46da-f26a-4f88-85e2-c6dc0c305a7a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5\" (UID: \"74bb46da-f26a-4f88-85e2-c6dc0c305a7a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5"
Oct 30 22:34:23.669263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.663174    4889 generic.go:356] "Generic (PLEG): container finished" podID="03c4d421-e943-45a1-91f9-71b9ec78daba" containerID="b8fef9f31a03651c323c69663c5892737e15772981e62c33e0369b589abe9c00" exitCode=0
Oct 30 22:34:23.669263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.663496    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w" event={"ID":"03c4d421-e943-45a1-91f9-71b9ec78daba","Type":"ContainerDied","Data":"b8fef9f31a03651c323c69663c5892737e15772981e62c33e0369b589abe9c00"}
Oct 30 22:34:23.669263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.663571    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w" event={"ID":"03c4d421-e943-45a1-91f9-71b9ec78daba","Type":"ContainerDied","Data":"6d4e9617abe6b8dc8cbdf8384bde9763c578ac36ed20a9a5460c2644370c2794"}
Oct 30 22:34:23.669263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.663610    4889 scope.go:122] "RemoveContainer" containerID="b8fef9f31a03651c323c69663c5892737e15772981e62c33e0369b589abe9c00"
Oct 30 22:34:23.669263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.663874    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w"
Oct 30 22:34:23.690309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.685710    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlx4b\" (UniqueName: \"kubernetes.io/projected/74bb46da-f26a-4f88-85e2-c6dc0c305a7a-kube-api-access-tlx4b\") pod \"payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5\" (UID: \"74bb46da-f26a-4f88-85e2-c6dc0c305a7a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5"
Oct 30 22:34:23.690309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.686007    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5"
Oct 30 22:34:23.739302 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.732620    4889 generic.go:356] "Generic (PLEG): container finished" podID="5c0067cb-bba0-4ae1-9e07-3412034dac9b" containerID="bc2b975ea14f5f2dc9df853978508c1f9fac9aa3b9a8e20d38d7546f0b1385a5" exitCode=0
Oct 30 22:34:23.739302 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.733980    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg"
Oct 30 22:34:23.749256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.741644    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg" event={"ID":"5c0067cb-bba0-4ae1-9e07-3412034dac9b","Type":"ContainerDied","Data":"bc2b975ea14f5f2dc9df853978508c1f9fac9aa3b9a8e20d38d7546f0b1385a5"}
Oct 30 22:34:23.749256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.741749    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg" event={"ID":"5c0067cb-bba0-4ae1-9e07-3412034dac9b","Type":"ContainerDied","Data":"58b5dfdaea435a9d65196308d82e6649d555d798dd2a84b71c6978ca5fa731de"}
Oct 30 22:34:23.866725 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.866700    4889 scope.go:122] "RemoveContainer" containerID="b8fef9f31a03651c323c69663c5892737e15772981e62c33e0369b589abe9c00"
Oct 30 22:34:23.867162 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:34:23.867133    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8fef9f31a03651c323c69663c5892737e15772981e62c33e0369b589abe9c00\": container with ID starting with b8fef9f31a03651c323c69663c5892737e15772981e62c33e0369b589abe9c00 not found: ID does not exist" containerID="b8fef9f31a03651c323c69663c5892737e15772981e62c33e0369b589abe9c00"
Oct 30 22:34:23.867306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.867176    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8fef9f31a03651c323c69663c5892737e15772981e62c33e0369b589abe9c00"} err="failed to get container status \"b8fef9f31a03651c323c69663c5892737e15772981e62c33e0369b589abe9c00\": rpc error: code = NotFound desc = could not find container \"b8fef9f31a03651c323c69663c5892737e15772981e62c33e0369b589abe9c00\": container with ID starting with b8fef9f31a03651c323c69663c5892737e15772981e62c33e0369b589abe9c00 not found: ID does not exist"
Oct 30 22:34:23.867306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.867201    4889 scope.go:122] "RemoveContainer" containerID="bc2b975ea14f5f2dc9df853978508c1f9fac9aa3b9a8e20d38d7546f0b1385a5"
Oct 30 22:34:23.982555 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.982517    4889 scope.go:122] "RemoveContainer" containerID="bc2b975ea14f5f2dc9df853978508c1f9fac9aa3b9a8e20d38d7546f0b1385a5"
Oct 30 22:34:23.984451 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:34:23.984422    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc2b975ea14f5f2dc9df853978508c1f9fac9aa3b9a8e20d38d7546f0b1385a5\": container with ID starting with bc2b975ea14f5f2dc9df853978508c1f9fac9aa3b9a8e20d38d7546f0b1385a5 not found: ID does not exist" containerID="bc2b975ea14f5f2dc9df853978508c1f9fac9aa3b9a8e20d38d7546f0b1385a5"
Oct 30 22:34:23.984678 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:23.984653    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc2b975ea14f5f2dc9df853978508c1f9fac9aa3b9a8e20d38d7546f0b1385a5"} err="failed to get container status \"bc2b975ea14f5f2dc9df853978508c1f9fac9aa3b9a8e20d38d7546f0b1385a5\": rpc error: code = NotFound desc = could not find container \"bc2b975ea14f5f2dc9df853978508c1f9fac9aa3b9a8e20d38d7546f0b1385a5\": container with ID starting with bc2b975ea14f5f2dc9df853978508c1f9fac9aa3b9a8e20d38d7546f0b1385a5 not found: ID does not exist"
Oct 30 22:34:24.005309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:24.005267    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w"]
Oct 30 22:34:24.027264 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:24.009499    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7a1ee7-55c68b95cf-s428w"]
Oct 30 22:34:24.049532 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:24.049494    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg"]
Oct 30 22:34:24.054021 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:24.053994    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7a1ee7-7dbbd77679-kbzmg"]
Oct 30 22:34:24.097738 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:24.097705    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 22:34:24.097894 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:24.097758    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z" podUID="8b34996e-28a0-4cbf-b0fa-4615ad9fccaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 22:34:24.198240 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:24.198147    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5"]
Oct 30 22:34:24.257442 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:24.257384    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:24.939052 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:24.939015    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03c4d421-e943-45a1-91f9-71b9ec78daba" path="/var/lib/kubelet/pods/03c4d421-e943-45a1-91f9-71b9ec78daba/volumes"
Oct 30 22:34:24.940829 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:24.940805    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c0067cb-bba0-4ae1-9e07-3412034dac9b" path="/var/lib/kubelet/pods/5c0067cb-bba0-4ae1-9e07-3412034dac9b/volumes"
Oct 30 22:34:24.941644 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:24.941605    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5" event={"ID":"74bb46da-f26a-4f88-85e2-c6dc0c305a7a","Type":"ContainerStarted","Data":"5a7c6fa0ec38f2346225df97d9b49421677a2b347d0a5a1d99727f4b6374de7b"}
Oct 30 22:34:24.941974 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:24.941940    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5"
Oct 30 22:34:24.942119 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:24.942102    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5" event={"ID":"74bb46da-f26a-4f88-85e2-c6dc0c305a7a","Type":"ContainerStarted","Data":"963f3d3015fa2f90468691f49fa283992430996336fd0945ad5faec84eb3e81a"}
Oct 30 22:34:24.955347 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:24.947654    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5" podStartSLOduration=1.947634518 podStartE2EDuration="1.947634518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:24.928495653 +0000 UTC m=+1650.284861968" watchObservedRunningTime="2026-10-30 22:34:24.947634518 +0000 UTC m=+1650.304000833"
Oct 30 22:34:25.100264 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:25.097817    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 22:34:25.100264 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:25.097922    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z" podUID="8b34996e-28a0-4cbf-b0fa-4615ad9fccaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 22:34:25.241642 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:25.241606    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:25.429303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:25.416748    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:25.635863 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:25.635756    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:25.939482 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:25.939376    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:26.097818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:26.097780    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 22:34:26.097974 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:26.097838    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z" podUID="8b34996e-28a0-4cbf-b0fa-4615ad9fccaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 22:34:27.102697 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:27.102665    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:34:27.103204 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:27.103102    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:34:27.103774 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:27.103751    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:34:27.946805 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:27.946768    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:29.531628 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:29.531586    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl"]
Oct 30 22:34:29.533035 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:29.533010    4889 cpu_manager.go:414] "containerMap: removing container" podUID="5c0067cb-bba0-4ae1-9e07-3412034dac9b" containerName="payload-processing"
Oct 30 22:34:29.533098 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:29.533039    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c0067cb-bba0-4ae1-9e07-3412034dac9b" containerName="payload-processing"
Oct 30 22:34:29.533098 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:29.533067    4889 cpu_manager.go:414] "containerMap: removing container" podUID="03c4d421-e943-45a1-91f9-71b9ec78daba" containerName="payload-pre-processing"
Oct 30 22:34:29.533098 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:29.533075    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03c4d421-e943-45a1-91f9-71b9ec78daba" containerName="payload-pre-processing"
Oct 30 22:34:29.533312 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:29.533298    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c0067cb-bba0-4ae1-9e07-3412034dac9b" containerName="payload-processing"
Oct 30 22:34:29.533354 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:29.533318    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="03c4d421-e943-45a1-91f9-71b9ec78daba" containerName="payload-pre-processing"
Oct 30 22:34:29.539337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:29.539313    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl"
Oct 30 22:34:29.555097 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:29.555068    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl"]
Oct 30 22:34:29.558015 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:29.557992    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-bdd94b-serving-cert\""
Oct 30 22:34:29.665381 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:29.665333    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ph22\" (UniqueName: \"kubernetes.io/projected/9fe49ab4-341a-4c8a-b993-6885741c01ae-kube-api-access-5ph22\") pod \"maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl\" (UID: \"9fe49ab4-341a-4c8a-b993-6885741c01ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl"
Oct 30 22:34:29.665920 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:29.665885    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fe49ab4-341a-4c8a-b993-6885741c01ae-maas-api-tls\") pod \"maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl\" (UID: \"9fe49ab4-341a-4c8a-b993-6885741c01ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl"
Oct 30 22:34:29.666098 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:29.666080    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe49ab4-341a-4c8a-b993-6885741c01ae-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl\" (UID: \"9fe49ab4-341a-4c8a-b993-6885741c01ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl"
Oct 30 22:34:29.666172 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:29.666137    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fe49ab4-341a-4c8a-b993-6885741c01ae-metrics-certs\") pod \"maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl\" (UID: \"9fe49ab4-341a-4c8a-b993-6885741c01ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl"
Oct 30 22:34:29.767286 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:29.767202    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe49ab4-341a-4c8a-b993-6885741c01ae-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl\" (UID: \"9fe49ab4-341a-4c8a-b993-6885741c01ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl"
Oct 30 22:34:29.767486 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:29.767319    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fe49ab4-341a-4c8a-b993-6885741c01ae-metrics-certs\") pod \"maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl\" (UID: \"9fe49ab4-341a-4c8a-b993-6885741c01ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl"
Oct 30 22:34:29.767486 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:29.767389    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ph22\" (UniqueName: \"kubernetes.io/projected/9fe49ab4-341a-4c8a-b993-6885741c01ae-kube-api-access-5ph22\") pod \"maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl\" (UID: \"9fe49ab4-341a-4c8a-b993-6885741c01ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl"
Oct 30 22:34:29.767625 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:29.767573    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:29.767685 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:29.767578    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fe49ab4-341a-4c8a-b993-6885741c01ae-maas-api-tls\") pod \"maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl\" (UID: \"9fe49ab4-341a-4c8a-b993-6885741c01ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl"
Oct 30 22:34:29.767685 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:34:29.767650    4889 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-serving-cert: secret "maas-api-e2e-inf-a-bdd94b-serving-cert" not found
Oct 30 22:34:29.767855 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:34:29.767724    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fe49ab4-341a-4c8a-b993-6885741c01ae-maas-api-tls podName:9fe49ab4-341a-4c8a-b993-6885741c01ae nodeName:}" failed. No retries permitted until 2026-10-30 22:34:30.267709245 +0000 UTC m=+1655.624075552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9fe49ab4-341a-4c8a-b993-6885741c01ae-maas-api-tls") pod "maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl" (UID: "9fe49ab4-341a-4c8a-b993-6885741c01ae") : secret "maas-api-e2e-inf-a-bdd94b-serving-cert" not found
Oct 30 22:34:29.768001 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:29.767980    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe49ab4-341a-4c8a-b993-6885741c01ae-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl\" (UID: \"9fe49ab4-341a-4c8a-b993-6885741c01ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl"
Oct 30 22:34:29.770010 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:29.769983    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fe49ab4-341a-4c8a-b993-6885741c01ae-metrics-certs\") pod \"maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl\" (UID: \"9fe49ab4-341a-4c8a-b993-6885741c01ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl"
Oct 30 22:34:29.786607 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:29.786544    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ph22\" (UniqueName: \"kubernetes.io/projected/9fe49ab4-341a-4c8a-b993-6885741c01ae-kube-api-access-5ph22\") pod \"maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl\" (UID: \"9fe49ab4-341a-4c8a-b993-6885741c01ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl"
Oct 30 22:34:29.935792 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:29.935756    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:30.154462 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:30.154357    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:30.281446 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:30.281413    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fe49ab4-341a-4c8a-b993-6885741c01ae-maas-api-tls\") pod \"maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl\" (UID: \"9fe49ab4-341a-4c8a-b993-6885741c01ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl"
Oct 30 22:34:30.283962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:30.283942    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fe49ab4-341a-4c8a-b993-6885741c01ae-maas-api-tls\") pod \"maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl\" (UID: \"9fe49ab4-341a-4c8a-b993-6885741c01ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl"
Oct 30 22:34:30.458761 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:30.458635    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl"
Oct 30 22:34:30.955311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:30.955277    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl"]
Oct 30 22:34:30.962507 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:34:30.962474    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe49ab4_341a_4c8a_b993_6885741c01ae.slice/crio-39f7036807225737d82070f495ca899eb597b29208e0969de60f1089120d48df WatchSource:0}: Error finding container 39f7036807225737d82070f495ca899eb597b29208e0969de60f1089120d48df: Status 404 returned error can't find the container with id 39f7036807225737d82070f495ca899eb597b29208e0969de60f1089120d48df
Oct 30 22:34:31.764475 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:31.764436    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:32.021906 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:32.021817    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl" event={"ID":"9fe49ab4-341a-4c8a-b993-6885741c01ae","Type":"ContainerStarted","Data":"76b84a5c7a008678f5a8a35d499897af7a9d1fbd3cfd120547c1cba95ff488ed"}
Oct 30 22:34:32.022510 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:32.022469    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl" event={"ID":"9fe49ab4-341a-4c8a-b993-6885741c01ae","Type":"ContainerStarted","Data":"39f7036807225737d82070f495ca899eb597b29208e0969de60f1089120d48df"}
Oct 30 22:34:32.024440 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:32.024422    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl"
Oct 30 22:34:32.073844 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:32.069895    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl" podStartSLOduration=2.772309463 podStartE2EDuration="3.069873858s" totalImagesPullingTime="297.564395ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:32.06865604 +0000 UTC m=+1657.425022358" watchObservedRunningTime="2026-10-30 22:34:32.069873858 +0000 UTC m=+1657.426240175"
Oct 30 22:34:32.592570 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:32.592156    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:32.775325 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:32.775287    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:32.827098 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:32.827062    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"]
Oct 30 22:34:32.827402 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:32.827376    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq" podUID="c81505a9-d460-44ad-ac11-0195c6f8bce5" containerName="istio-proxy" containerID="cri-o://c94038f6b9536dfb03ef18c1470371191614a9103ee28f693a26190ec427ddfd" gracePeriod=30
Oct 30 22:34:33.005492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:33.005456    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:33.845684 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:33.844389    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:35.726323 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:35.722568    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:35.890327 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:35.890300    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5"
Oct 30 22:34:37.751245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:37.750549    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:38.132328 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.132295    4889 generic.go:356] "Generic (PLEG): container finished" podID="c81505a9-d460-44ad-ac11-0195c6f8bce5" containerID="c94038f6b9536dfb03ef18c1470371191614a9103ee28f693a26190ec427ddfd" exitCode=0
Oct 30 22:34:38.132498 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.132368    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq" event={"ID":"c81505a9-d460-44ad-ac11-0195c6f8bce5","Type":"ContainerDied","Data":"c94038f6b9536dfb03ef18c1470371191614a9103ee28f693a26190ec427ddfd"}
Oct 30 22:34:38.132498 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.132432    4889 scope.go:122] "RemoveContainer" containerID="e8ec72c89a7d46dcb289290ddf46b65a1985f6f3d594536b2425667a213a49bd"
Oct 30 22:34:38.166559 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.166533    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:34:38.255579 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.255542    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-workload-socket\") pod \"c81505a9-d460-44ad-ac11-0195c6f8bce5\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") "
Oct 30 22:34:38.255770 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.255604    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-token\" (UniqueName: \"kubernetes.io/projected/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-token\") pod \"c81505a9-d460-44ad-ac11-0195c6f8bce5\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") "
Oct 30 22:34:38.255770 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.255707    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-podinfo\") pod \"c81505a9-d460-44ad-ac11-0195c6f8bce5\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") "
Oct 30 22:34:38.255890 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.255777    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-workload-certs\") pod \"c81505a9-d460-44ad-ac11-0195c6f8bce5\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") "
Oct 30 22:34:38.255890 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.255798    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-workload-socket" pod "c81505a9-d460-44ad-ac11-0195c6f8bce5" (UID: "c81505a9-d460-44ad-ac11-0195c6f8bce5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:38.255890 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.255872    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-credential-socket\") pod \"c81505a9-d460-44ad-ac11-0195c6f8bce5\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") "
Oct 30 22:34:38.256037 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.255989    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-data\") pod \"c81505a9-d460-44ad-ac11-0195c6f8bce5\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") "
Oct 30 22:34:38.256091 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.256059    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c81505a9-d460-44ad-ac11-0195c6f8bce5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81505a9-d460-44ad-ac11-0195c6f8bce5-istiod-ca-cert\") pod \"c81505a9-d460-44ad-ac11-0195c6f8bce5\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") "
Oct 30 22:34:38.256153 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.256114    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81505a9-d460-44ad-ac11-0195c6f8bce5-kube-api-access-zp49r\" (UniqueName: \"kubernetes.io/projected/c81505a9-d460-44ad-ac11-0195c6f8bce5-kube-api-access-zp49r\") pod \"c81505a9-d460-44ad-ac11-0195c6f8bce5\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") "
Oct 30 22:34:38.256214 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.256187    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-workload-certs" pod "c81505a9-d460-44ad-ac11-0195c6f8bce5" (UID: "c81505a9-d460-44ad-ac11-0195c6f8bce5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:38.256304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.256215    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-envoy\") pod \"c81505a9-d460-44ad-ac11-0195c6f8bce5\" (UID: \"c81505a9-d460-44ad-ac11-0195c6f8bce5\") "
Oct 30 22:34:38.256669 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.256402    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-data" pod "c81505a9-d460-44ad-ac11-0195c6f8bce5" (UID: "c81505a9-d460-44ad-ac11-0195c6f8bce5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:38.257078 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.256824    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-credential-socket" pod "c81505a9-d460-44ad-ac11-0195c6f8bce5" (UID: "c81505a9-d460-44ad-ac11-0195c6f8bce5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:38.257175 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.257096    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:34:38.257175 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.257115    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:34:38.257175 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.257129    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:34:38.257175 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.257143    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:34:38.257986 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.257964    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c81505a9-d460-44ad-ac11-0195c6f8bce5-istiod-ca-cert" pod "c81505a9-d460-44ad-ac11-0195c6f8bce5" (UID: "c81505a9-d460-44ad-ac11-0195c6f8bce5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:38.258439 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.258411    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-podinfo" pod "c81505a9-d460-44ad-ac11-0195c6f8bce5" (UID: "c81505a9-d460-44ad-ac11-0195c6f8bce5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:34:38.258652 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.258627    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-token" pod "c81505a9-d460-44ad-ac11-0195c6f8bce5" (UID: "c81505a9-d460-44ad-ac11-0195c6f8bce5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:38.259429 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.259404    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-envoy" pod "c81505a9-d460-44ad-ac11-0195c6f8bce5" (UID: "c81505a9-d460-44ad-ac11-0195c6f8bce5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:38.260372 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.260345    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81505a9-d460-44ad-ac11-0195c6f8bce5-kube-api-access-zp49r" pod "c81505a9-d460-44ad-ac11-0195c6f8bce5" (UID: "c81505a9-d460-44ad-ac11-0195c6f8bce5"). InnerVolumeSpecName "kube-api-access-zp49r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:38.359237 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.359198    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81505a9-d460-44ad-ac11-0195c6f8bce5-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:34:38.359237 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.359245    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp49r\" (UniqueName: \"kubernetes.io/projected/c81505a9-d460-44ad-ac11-0195c6f8bce5-kube-api-access-zp49r\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:34:38.359444 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.359257    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:34:38.359444 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.359265    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:34:38.359444 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:38.359273    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81505a9-d460-44ad-ac11-0195c6f8bce5-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:34:39.085318 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:39.085239    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl"
Oct 30 22:34:39.147177 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:39.147137    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq" event={"ID":"c81505a9-d460-44ad-ac11-0195c6f8bce5","Type":"ContainerDied","Data":"60e72d7b26ae39b4c261a1d7e6142f44b500e593e1f54bcf8175a9cd392dc3b8"}
Oct 30 22:34:39.147177 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:39.147170    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"
Oct 30 22:34:39.147460 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:39.147213    4889 scope.go:122] "RemoveContainer" containerID="c94038f6b9536dfb03ef18c1470371191614a9103ee28f693a26190ec427ddfd"
Oct 30 22:34:39.253124 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:39.253064    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"]
Oct 30 22:34:39.263320 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:39.263290    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-de31dd-openshift-default-56f5bd455-d8vcq"]
Oct 30 22:34:39.509253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:39.507966    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:40.304418 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:34:40.304385    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:34:40.778485 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:40.778446    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c81505a9-d460-44ad-ac11-0195c6f8bce5" path="/var/lib/kubelet/pods/c81505a9-d460-44ad-ac11-0195c6f8bce5/volumes"
Oct 30 22:34:41.321093 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:41.321053    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:43.060694 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:43.060654    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:44.222433 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:44.222401    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x"]
Oct 30 22:34:44.253686 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:44.253652    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv"]
Oct 30 22:34:44.527153 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:44.527118    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:44.698908 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:44.698867    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:44.903754 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:44.894402    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:46.482266 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:46.480353    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:48.136605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:48.136568    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:48.341286 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:48.339545    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:49.223002 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:49.222962    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x" podUID="337ddd2a-f009-47bd-a214-9ca32e15c9f9" containerName="payload-processing" containerID="cri-o://dd0718a5be3afc2b162a03fa0e252a5f7f7a2512a6dc8ffe229591d42eb3e2d6" gracePeriod=25
Oct 30 22:34:49.254682 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:49.254649    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv" podUID="31998a4b-a2ae-4128-8a93-916f9247f06d" containerName="payload-pre-processing" containerID="cri-o://8b7be5263ac5151279d5331856f84001d8244240d82f53f99ba5dc9f1e6b03aa" gracePeriod=25
Oct 30 22:34:49.556844 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:49.556817    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x"
Oct 30 22:34:49.602180 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:49.602153    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv"
Oct 30 22:34:49.680930 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:49.680894    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/337ddd2a-f009-47bd-a214-9ca32e15c9f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/337ddd2a-f009-47bd-a214-9ca32e15c9f9-plugins-config-volume\") pod \"337ddd2a-f009-47bd-a214-9ca32e15c9f9\" (UID: \"337ddd2a-f009-47bd-a214-9ca32e15c9f9\") "
Oct 30 22:34:49.681275 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:49.680968    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31998a4b-a2ae-4128-8a93-916f9247f06d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31998a4b-a2ae-4128-8a93-916f9247f06d-plugins-config-volume\") pod \"31998a4b-a2ae-4128-8a93-916f9247f06d\" (UID: \"31998a4b-a2ae-4128-8a93-916f9247f06d\") "
Oct 30 22:34:49.681275 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:49.681109    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/337ddd2a-f009-47bd-a214-9ca32e15c9f9-kube-api-access-qwkth\" (UniqueName: \"kubernetes.io/projected/337ddd2a-f009-47bd-a214-9ca32e15c9f9-kube-api-access-qwkth\") pod \"337ddd2a-f009-47bd-a214-9ca32e15c9f9\" (UID: \"337ddd2a-f009-47bd-a214-9ca32e15c9f9\") "
Oct 30 22:34:49.681275 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:49.681219    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31998a4b-a2ae-4128-8a93-916f9247f06d-kube-api-access-qmnbl\" (UniqueName: \"kubernetes.io/projected/31998a4b-a2ae-4128-8a93-916f9247f06d-kube-api-access-qmnbl\") pod \"31998a4b-a2ae-4128-8a93-916f9247f06d\" (UID: \"31998a4b-a2ae-4128-8a93-916f9247f06d\") "
Oct 30 22:34:49.682734 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:49.682704    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31998a4b-a2ae-4128-8a93-916f9247f06d-plugins-config-volume" pod "31998a4b-a2ae-4128-8a93-916f9247f06d" (UID: "31998a4b-a2ae-4128-8a93-916f9247f06d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:49.683619 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:49.683595    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/337ddd2a-f009-47bd-a214-9ca32e15c9f9-plugins-config-volume" pod "337ddd2a-f009-47bd-a214-9ca32e15c9f9" (UID: "337ddd2a-f009-47bd-a214-9ca32e15c9f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:49.684852 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:49.684822    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31998a4b-a2ae-4128-8a93-916f9247f06d-kube-api-access-qmnbl" pod "31998a4b-a2ae-4128-8a93-916f9247f06d" (UID: "31998a4b-a2ae-4128-8a93-916f9247f06d"). InnerVolumeSpecName "kube-api-access-qmnbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:49.685999 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:49.685978    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337ddd2a-f009-47bd-a214-9ca32e15c9f9-kube-api-access-qwkth" pod "337ddd2a-f009-47bd-a214-9ca32e15c9f9" (UID: "337ddd2a-f009-47bd-a214-9ca32e15c9f9"). InnerVolumeSpecName "kube-api-access-qwkth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:49.784169 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:49.784093    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/337ddd2a-f009-47bd-a214-9ca32e15c9f9-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:34:49.784169 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:49.784121    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31998a4b-a2ae-4128-8a93-916f9247f06d-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:34:49.784169 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:49.784133    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwkth\" (UniqueName: \"kubernetes.io/projected/337ddd2a-f009-47bd-a214-9ca32e15c9f9-kube-api-access-qwkth\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:34:49.784169 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:49.784141    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmnbl\" (UniqueName: \"kubernetes.io/projected/31998a4b-a2ae-4128-8a93-916f9247f06d-kube-api-access-qmnbl\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:34:50.027555 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:50.027519    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:50.238107 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:50.238021    4889 generic.go:356] "Generic (PLEG): container finished" podID="31998a4b-a2ae-4128-8a93-916f9247f06d" containerID="8b7be5263ac5151279d5331856f84001d8244240d82f53f99ba5dc9f1e6b03aa" exitCode=0
Oct 30 22:34:50.238107 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:50.238073    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv"
Oct 30 22:34:50.238633 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:50.238099    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv" event={"ID":"31998a4b-a2ae-4128-8a93-916f9247f06d","Type":"ContainerDied","Data":"8b7be5263ac5151279d5331856f84001d8244240d82f53f99ba5dc9f1e6b03aa"}
Oct 30 22:34:50.238633 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:50.238172    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv" event={"ID":"31998a4b-a2ae-4128-8a93-916f9247f06d","Type":"ContainerDied","Data":"aa1974a6282567762edaed3d680c3def5cdedf50dcf008978695d7b6a3fbb320"}
Oct 30 22:34:50.238633 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:50.238202    4889 scope.go:122] "RemoveContainer" containerID="8b7be5263ac5151279d5331856f84001d8244240d82f53f99ba5dc9f1e6b03aa"
Oct 30 22:34:50.240428 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:50.240362    4889 generic.go:356] "Generic (PLEG): container finished" podID="337ddd2a-f009-47bd-a214-9ca32e15c9f9" containerID="dd0718a5be3afc2b162a03fa0e252a5f7f7a2512a6dc8ffe229591d42eb3e2d6" exitCode=0
Oct 30 22:34:50.240428 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:50.240401    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x" event={"ID":"337ddd2a-f009-47bd-a214-9ca32e15c9f9","Type":"ContainerDied","Data":"dd0718a5be3afc2b162a03fa0e252a5f7f7a2512a6dc8ffe229591d42eb3e2d6"}
Oct 30 22:34:50.240428 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:50.240417    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x"
Oct 30 22:34:50.240689 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:50.240453    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x" event={"ID":"337ddd2a-f009-47bd-a214-9ca32e15c9f9","Type":"ContainerDied","Data":"558778189b23e23340aff38789d1d5c2a6336d4206dba3928447ca20d4444fa2"}
Oct 30 22:34:50.254782 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:50.254767    4889 scope.go:122] "RemoveContainer" containerID="8b7be5263ac5151279d5331856f84001d8244240d82f53f99ba5dc9f1e6b03aa"
Oct 30 22:34:50.255030 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:34:50.255015    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b7be5263ac5151279d5331856f84001d8244240d82f53f99ba5dc9f1e6b03aa\": container with ID starting with 8b7be5263ac5151279d5331856f84001d8244240d82f53f99ba5dc9f1e6b03aa not found: ID does not exist" containerID="8b7be5263ac5151279d5331856f84001d8244240d82f53f99ba5dc9f1e6b03aa"
Oct 30 22:34:50.255079 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:50.255037    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b7be5263ac5151279d5331856f84001d8244240d82f53f99ba5dc9f1e6b03aa"} err="failed to get container status \"8b7be5263ac5151279d5331856f84001d8244240d82f53f99ba5dc9f1e6b03aa\": rpc error: code = NotFound desc = could not find container \"8b7be5263ac5151279d5331856f84001d8244240d82f53f99ba5dc9f1e6b03aa\": container with ID starting with 8b7be5263ac5151279d5331856f84001d8244240d82f53f99ba5dc9f1e6b03aa not found: ID does not exist"
Oct 30 22:34:50.255079 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:50.255051    4889 scope.go:122] "RemoveContainer" containerID="dd0718a5be3afc2b162a03fa0e252a5f7f7a2512a6dc8ffe229591d42eb3e2d6"
Oct 30 22:34:50.266387 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:50.266371    4889 scope.go:122] "RemoveContainer" containerID="dd0718a5be3afc2b162a03fa0e252a5f7f7a2512a6dc8ffe229591d42eb3e2d6"
Oct 30 22:34:50.266642 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:34:50.266624    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd0718a5be3afc2b162a03fa0e252a5f7f7a2512a6dc8ffe229591d42eb3e2d6\": container with ID starting with dd0718a5be3afc2b162a03fa0e252a5f7f7a2512a6dc8ffe229591d42eb3e2d6 not found: ID does not exist" containerID="dd0718a5be3afc2b162a03fa0e252a5f7f7a2512a6dc8ffe229591d42eb3e2d6"
Oct 30 22:34:50.266688 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:50.266649    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd0718a5be3afc2b162a03fa0e252a5f7f7a2512a6dc8ffe229591d42eb3e2d6"} err="failed to get container status \"dd0718a5be3afc2b162a03fa0e252a5f7f7a2512a6dc8ffe229591d42eb3e2d6\": rpc error: code = NotFound desc = could not find container \"dd0718a5be3afc2b162a03fa0e252a5f7f7a2512a6dc8ffe229591d42eb3e2d6\": container with ID starting with dd0718a5be3afc2b162a03fa0e252a5f7f7a2512a6dc8ffe229591d42eb3e2d6 not found: ID does not exist"
Oct 30 22:34:50.309923 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:50.309889    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv"]
Oct 30 22:34:50.315433 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:50.315406    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8b3e43-57f97b8bcf-x88zv"]
Oct 30 22:34:50.351713 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:50.351682    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x"]
Oct 30 22:34:50.357449 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:50.357422    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8b3e43-685659576c-mxc2x"]
Oct 30 22:34:50.777565 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:50.777534    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31998a4b-a2ae-4128-8a93-916f9247f06d" path="/var/lib/kubelet/pods/31998a4b-a2ae-4128-8a93-916f9247f06d/volumes"
Oct 30 22:34:50.778111 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:50.778092    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="337ddd2a-f009-47bd-a214-9ca32e15c9f9" path="/var/lib/kubelet/pods/337ddd2a-f009-47bd-a214-9ca32e15c9f9/volumes"
Oct 30 22:34:51.794884 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:51.794842    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:53.752414 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:53.752254    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:55.932651 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:55.932613    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:57.880538 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:57.880505    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:59.663491 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:34:59.663454    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:01.279581 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:01.279544    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:02.044132 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.044095    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c"]
Oct 30 22:35:02.045039 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.045020    4889 cpu_manager.go:414] "containerMap: removing container" podUID="c81505a9-d460-44ad-ac11-0195c6f8bce5" containerName="istio-proxy"
Oct 30 22:35:02.045142 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.045040    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c81505a9-d460-44ad-ac11-0195c6f8bce5" containerName="istio-proxy"
Oct 30 22:35:02.045142 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.045062    4889 cpu_manager.go:414] "containerMap: removing container" podUID="c81505a9-d460-44ad-ac11-0195c6f8bce5" containerName="istio-proxy"
Oct 30 22:35:02.045142 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.045067    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c81505a9-d460-44ad-ac11-0195c6f8bce5" containerName="istio-proxy"
Oct 30 22:35:02.045142 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.045076    4889 cpu_manager.go:414] "containerMap: removing container" podUID="31998a4b-a2ae-4128-8a93-916f9247f06d" containerName="payload-pre-processing"
Oct 30 22:35:02.045142 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.045082    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31998a4b-a2ae-4128-8a93-916f9247f06d" containerName="payload-pre-processing"
Oct 30 22:35:02.045142 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.045088    4889 cpu_manager.go:414] "containerMap: removing container" podUID="337ddd2a-f009-47bd-a214-9ca32e15c9f9" containerName="payload-processing"
Oct 30 22:35:02.045142 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.045092    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="337ddd2a-f009-47bd-a214-9ca32e15c9f9" containerName="payload-processing"
Oct 30 22:35:02.045470 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.045199    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="c81505a9-d460-44ad-ac11-0195c6f8bce5" containerName="istio-proxy"
Oct 30 22:35:02.045470 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.045209    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="31998a4b-a2ae-4128-8a93-916f9247f06d" containerName="payload-pre-processing"
Oct 30 22:35:02.045470 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.045216    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="337ddd2a-f009-47bd-a214-9ca32e15c9f9" containerName="payload-processing"
Oct 30 22:35:02.049354 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.049331    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c"
Oct 30 22:35:02.069101 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.069065    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c"]
Oct 30 22:35:02.072686 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.072651    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f9d87b-serving-cert\""
Oct 30 22:35:02.088877 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.088835    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/625280d4-8b36-4188-a38d-725f4cdd55e3-metrics-certs\") pod \"maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c\" (UID: \"625280d4-8b36-4188-a38d-725f4cdd55e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c"
Oct 30 22:35:02.089266 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.089245    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625280d4-8b36-4188-a38d-725f4cdd55e3-maas-api-tls\") pod \"maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c\" (UID: \"625280d4-8b36-4188-a38d-725f4cdd55e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c"
Oct 30 22:35:02.089467 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.089450    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkkb5\" (UniqueName: \"kubernetes.io/projected/625280d4-8b36-4188-a38d-725f4cdd55e3-kube-api-access-nkkb5\") pod \"maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c\" (UID: \"625280d4-8b36-4188-a38d-725f4cdd55e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c"
Oct 30 22:35:02.089584 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.089571    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/625280d4-8b36-4188-a38d-725f4cdd55e3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c\" (UID: \"625280d4-8b36-4188-a38d-725f4cdd55e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c"
Oct 30 22:35:02.190794 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.190758    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/625280d4-8b36-4188-a38d-725f4cdd55e3-metrics-certs\") pod \"maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c\" (UID: \"625280d4-8b36-4188-a38d-725f4cdd55e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c"
Oct 30 22:35:02.190975 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.190865    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625280d4-8b36-4188-a38d-725f4cdd55e3-maas-api-tls\") pod \"maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c\" (UID: \"625280d4-8b36-4188-a38d-725f4cdd55e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c"
Oct 30 22:35:02.190975 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.190922    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkkb5\" (UniqueName: \"kubernetes.io/projected/625280d4-8b36-4188-a38d-725f4cdd55e3-kube-api-access-nkkb5\") pod \"maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c\" (UID: \"625280d4-8b36-4188-a38d-725f4cdd55e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c"
Oct 30 22:35:02.190975 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.190954    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/625280d4-8b36-4188-a38d-725f4cdd55e3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c\" (UID: \"625280d4-8b36-4188-a38d-725f4cdd55e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c"
Oct 30 22:35:02.191137 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:35:02.191010    4889 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-serving-cert: secret "maas-api-e2e-inf-b-f9d87b-serving-cert" not found
Oct 30 22:35:02.191137 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:35:02.191080    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/625280d4-8b36-4188-a38d-725f4cdd55e3-maas-api-tls podName:625280d4-8b36-4188-a38d-725f4cdd55e3 nodeName:}" failed. No retries permitted until 2026-10-30 22:35:02.691058595 +0000 UTC m=+1688.047424894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/625280d4-8b36-4188-a38d-725f4cdd55e3-maas-api-tls") pod "maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c" (UID: "625280d4-8b36-4188-a38d-725f4cdd55e3") : secret "maas-api-e2e-inf-b-f9d87b-serving-cert" not found
Oct 30 22:35:02.191741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.191719    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/625280d4-8b36-4188-a38d-725f4cdd55e3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c\" (UID: \"625280d4-8b36-4188-a38d-725f4cdd55e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c"
Oct 30 22:35:02.194875 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.194829    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/625280d4-8b36-4188-a38d-725f4cdd55e3-metrics-certs\") pod \"maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c\" (UID: \"625280d4-8b36-4188-a38d-725f4cdd55e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c"
Oct 30 22:35:02.210996 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.210966    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkkb5\" (UniqueName: \"kubernetes.io/projected/625280d4-8b36-4188-a38d-725f4cdd55e3-kube-api-access-nkkb5\") pod \"maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c\" (UID: \"625280d4-8b36-4188-a38d-725f4cdd55e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c"
Oct 30 22:35:02.318602 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.318512    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:02.497769 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.497732    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:02.662800 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.662704    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:02.725269 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.724860    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625280d4-8b36-4188-a38d-725f4cdd55e3-maas-api-tls\") pod \"maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c\" (UID: \"625280d4-8b36-4188-a38d-725f4cdd55e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c"
Oct 30 22:35:02.733250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.731409    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625280d4-8b36-4188-a38d-725f4cdd55e3-maas-api-tls\") pod \"maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c\" (UID: \"625280d4-8b36-4188-a38d-725f4cdd55e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c"
Oct 30 22:35:02.823473 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.823433    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:02.969370 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:02.969276    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c"
Oct 30 22:35:03.251948 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:03.251913    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:03.337622 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:35:03.337586    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625280d4_8b36_4188_a38d_725f4cdd55e3.slice/crio-c5d4991c73787b440e823099dd1b36e9532227120c6aaeaad24719db8da9daeb WatchSource:0}: Error finding container c5d4991c73787b440e823099dd1b36e9532227120c6aaeaad24719db8da9daeb: Status 404 returned error can't find the container with id c5d4991c73787b440e823099dd1b36e9532227120c6aaeaad24719db8da9daeb
Oct 30 22:35:03.338134 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:03.338047    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c"]
Oct 30 22:35:03.339463 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:03.339442    4889 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:35:03.521429 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:03.519381    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c" event={"ID":"625280d4-8b36-4188-a38d-725f4cdd55e3","Type":"ContainerStarted","Data":"c5d4991c73787b440e823099dd1b36e9532227120c6aaeaad24719db8da9daeb"}
Oct 30 22:35:04.358248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:04.354878    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:04.529060 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:04.529019    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c" event={"ID":"625280d4-8b36-4188-a38d-725f4cdd55e3","Type":"ContainerStarted","Data":"cbac7be9e76cf849f6e2ca31a3d5a6c2d8ca7b8ee32b609a5b454859cd9aeb7a"}
Oct 30 22:35:04.529277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:04.529257    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c"
Oct 30 22:35:04.573423 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:04.573365    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c" podStartSLOduration=2.097491134 podStartE2EDuration="2.573347145s" totalImagesPullingTime="475.856011ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:04.567906392 +0000 UTC m=+1689.924272719" watchObservedRunningTime="2026-10-30 22:35:04.573347145 +0000 UTC m=+1689.929713460"
Oct 30 22:35:06.372004 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:06.371968    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:07.554575 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:07.554536    4889 generic.go:356] "Generic (PLEG): container finished" podID="55f78a17-ccc6-4f75-902e-e048a59e6c6a" containerID="3147bbc32cb944f07e6d215f8d6c35dc3cabd7134ec9f53e91b25b94fd68bf26" exitCode=0
Oct 30 22:35:07.554992 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:07.554602    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p" event={"ID":"55f78a17-ccc6-4f75-902e-e048a59e6c6a","Type":"ContainerDied","Data":"3147bbc32cb944f07e6d215f8d6c35dc3cabd7134ec9f53e91b25b94fd68bf26"}
Oct 30 22:35:07.555081 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:07.555067    4889 scope.go:122] "RemoveContainer" containerID="3147bbc32cb944f07e6d215f8d6c35dc3cabd7134ec9f53e91b25b94fd68bf26"
Oct 30 22:35:07.556987 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:07.556954    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:35:07.557109 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:07.557041    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:35:07.557174 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:07.557111    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:35:08.130254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:08.128811    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:08.565785 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:08.565744    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p" event={"ID":"55f78a17-ccc6-4f75-902e-e048a59e6c6a","Type":"ContainerStarted","Data":"ac7e91577023d9c75d8bc84c680a58f6f90f290bdcebdc5e50ce75b934ec31f5"}
Oct 30 22:35:09.393490 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:09.393457    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:35:09.394622 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:09.394594    4889 patch_prober.go:29] interesting pod/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 22:35:09.394748 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:09.394646    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p" podUID="55f78a17-ccc6-4f75-902e-e048a59e6c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 22:35:09.606418 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:09.606385    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:10.300275 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:10.288954    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:10.406250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:10.405618    4889 patch_prober.go:29] interesting pod/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 22:35:10.406250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:10.405707    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p" podUID="55f78a17-ccc6-4f75-902e-e048a59e6c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 22:35:10.494556 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:35:10.492891    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:35:10.494556 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:10.494349    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:10.564668 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:10.564569    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c"
Oct 30 22:35:10.787246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:10.785086    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:11.001305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:10.977829    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:11.180387 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:11.180355    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:11.295656 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:11.295565    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:35:11.353811 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:11.353779    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:11.394019 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:11.393963    4889 patch_prober.go:29] interesting pod/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 22:35:11.394374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:11.394342    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p" podUID="55f78a17-ccc6-4f75-902e-e048a59e6c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 22:35:12.393795 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:12.393757    4889 patch_prober.go:29] interesting pod/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 22:35:12.394353 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:12.393809    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p" podUID="55f78a17-ccc6-4f75-902e-e048a59e6c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 22:35:13.367921 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:13.365241    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:13.397268 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:13.397213    4889 patch_prober.go:29] interesting pod/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 22:35:13.402133 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:13.400943    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p" podUID="55f78a17-ccc6-4f75-902e-e048a59e6c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 22:35:14.394181 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:14.394145    4889 patch_prober.go:29] interesting pod/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 22:35:14.394391 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:14.394214    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p" podUID="55f78a17-ccc6-4f75-902e-e048a59e6c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 22:35:15.612733 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:15.612702    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:35:15.614193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:15.614168    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:35:15.813938 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:15.813907    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:17.903434 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:17.903395    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:18.516025 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:18.515977    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:18.684528 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:18.684493    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:18.890310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:18.884909    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:19.671445 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:19.671413    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:19.892067 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:19.891994    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:20.030402 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.030368    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"]
Oct 30 22:35:20.031626 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.031601    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="c81505a9-d460-44ad-ac11-0195c6f8bce5" containerName="istio-proxy"
Oct 30 22:35:20.034691 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.034661    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.049352 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.049323    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-999470-openshift-default-dockercfg-6pnwb\""
Oct 30 22:35:20.070800 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.070568    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"]
Oct 30 22:35:20.082563 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.082529    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:20.111699 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.111656    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-data\") pod \"e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") " pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.111699 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.111697    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-podinfo\") pod \"e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") " pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.111990 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.111899    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-credential-socket\") pod \"e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") " pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.112060 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.112000    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-workload-socket\") pod \"e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") " pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.112119 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.112057    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istiod-ca-cert\") pod \"e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") " pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.112119 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.112095    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72r26\" (UniqueName: \"kubernetes.io/projected/b0758d58-495a-4f8e-a1f6-daaca5024f8c-kube-api-access-72r26\") pod \"e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") " pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.112344 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.112325    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-envoy\") pod \"e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") " pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.112429 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.112364    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-workload-certs\") pod \"e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") " pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.112523 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.112499    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-token\") pod \"e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") " pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.213678 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.213643    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-workload-certs\") pod \"e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") " pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.213895 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.213716    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-token\") pod \"e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") " pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.213895 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.213742    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-data\") pod \"e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") " pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.213895 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.213763    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-podinfo\") pod \"e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") " pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.213895 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.213846    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-credential-socket\") pod \"e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") " pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.214157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.213901    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-workload-socket\") pod \"e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") " pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.214157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.213967    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istiod-ca-cert\") pod \"e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") " pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.214157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.214020    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72r26\" (UniqueName: \"kubernetes.io/projected/b0758d58-495a-4f8e-a1f6-daaca5024f8c-kube-api-access-72r26\") pod \"e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") " pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.214157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.214101    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-workload-certs\") pod \"e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") " pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.214467 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.214273    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-envoy\") pod \"e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") " pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.214467 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.214300    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-credential-socket\") pod \"e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") " pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.214467 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.214418    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-workload-socket\") pod \"e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") " pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.214614 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.214564    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-data\") pod \"e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") " pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.214782 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.214758    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istiod-ca-cert\") pod \"e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") " pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.216783 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.216763    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-envoy\") pod \"e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") " pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.216951 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.216927    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-podinfo\") pod \"e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") " pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.297303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.293985    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72r26\" (UniqueName: \"kubernetes.io/projected/b0758d58-495a-4f8e-a1f6-daaca5024f8c-kube-api-access-72r26\") pod \"e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") " pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.297303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.294678    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-token\") pod \"e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") " pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.297303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.294046    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:20.351644 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.351356    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:20.473775 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.473740    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:20.645109 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.644948    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:20.886769 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.886735    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"]
Oct 30 22:35:20.887137 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:35:20.886837    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0758d58_495a_4f8e_a1f6_daaca5024f8c.slice/crio-8472ba818549691e0db686c068747027de469ebf40a64edbc39fcde42a22d6d2 WatchSource:0}: Error finding container 8472ba818549691e0db686c068747027de469ebf40a64edbc39fcde42a22d6d2: Status 404 returned error can't find the container with id 8472ba818549691e0db686c068747027de469ebf40a64edbc39fcde42a22d6d2
Oct 30 22:35:20.890240 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.890190    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:35:20.890344 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.890305    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:35:20.890344 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:20.890342    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:35:21.121251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:21.113185    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:21.608990 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:21.604611    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:21.711891 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:21.711838    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd" event={"ID":"b0758d58-495a-4f8e-a1f6-daaca5024f8c","Type":"ContainerStarted","Data":"1441a09a6ac5bf01a605cb2ea241faee3b13fc3c8da4ae0cdb0c0ba560b991b6"}
Oct 30 22:35:21.712076 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:21.711913    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd" event={"ID":"b0758d58-495a-4f8e-a1f6-daaca5024f8c","Type":"ContainerStarted","Data":"8472ba818549691e0db686c068747027de469ebf40a64edbc39fcde42a22d6d2"}
Oct 30 22:35:21.765134 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:21.765037    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd" podStartSLOduration=1.765017987 podStartE2EDuration="1.765017987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:21.763891987 +0000 UTC m=+1707.120258302" watchObservedRunningTime="2026-10-30 22:35:21.765017987 +0000 UTC m=+1707.121384306"
Oct 30 22:35:22.040422 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:22.040381    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:22.230249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:22.228067    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:22.353730 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:22.353651    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:22.360583 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:22.360541    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:22.799271 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:22.799217    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:22.799492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:22.799343    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:35:24.063431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:24.063392    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:24.496157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:24.496122    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:24.701982 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:24.701945    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:25.486741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:25.486710    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:25.687135 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:25.687094    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:26.262672 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:26.262642    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:26.459194 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:26.459155    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:28.175276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:28.175205    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:28.433530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:28.416900    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:28.630960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:28.629441    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:30.014967 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:30.014930    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:30.698621 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:30.698589    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:30.874526 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:30.874476    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:31.040303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:31.040181    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:31.336825 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:31.336730    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:32.741771 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:32.741680    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:34.587348 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:34.587272    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:36.163283 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:36.163243    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:36.855644 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:36.855600    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:37.025277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:37.025190    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:37.198252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:37.198141    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:37.554625 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:37.554583    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:38.715297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:38.715202    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:38.892382 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:38.892344    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:39.106034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:39.105992    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:39.453379 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:39.452581    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:40.305139 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:35:40.305101    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:35:40.948697 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:40.948628    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:42.944346 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:42.944310    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:44.627377 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:44.627343    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:44.921329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:44.919857    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:45.087318 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:45.087281    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:45.135735 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:45.135662    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"]
Oct 30 22:35:45.135967 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:45.135938    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp" podUID="be263888-3047-4ee6-88cf-0fe7a033f3e1" containerName="istio-proxy" containerID="cri-o://a030ba12d8f39b8aceeedc969b6fa04529aad4c194e41ad6ca4ba46b6ac026ec" gracePeriod=30
Oct 30 22:35:45.309038 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:45.309005    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:46.026832 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:46.026795    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:46.647976 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:46.647894    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:47.041010 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:47.040713    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:47.333299 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:47.333265    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:48.254280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:48.254238    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:49.546927 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:49.546888    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:49.685416 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:49.685356    4889 patch_prober.go:29] interesting pod/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:35:49.685618 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:49.685449    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp" podUID="be263888-3047-4ee6-88cf-0fe7a033f3e1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:35:49.748517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:49.748479    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf"]
Oct 30 22:35:49.780774 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:49.780740    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l"]
Oct 30 22:35:49.949129 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:49.948809    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:50.159166 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.159134    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:50.519579 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.487393    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:50.741803 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.741719    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:50.786734 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.786707    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:35:50.803976 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.803949    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-podinfo\") pod \"be263888-3047-4ee6-88cf-0fe7a033f3e1\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") "
Oct 30 22:35:50.804148 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.803991    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-workload-certs\") pod \"be263888-3047-4ee6-88cf-0fe7a033f3e1\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") "
Oct 30 22:35:50.804148 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.804020    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-token\" (UniqueName: \"kubernetes.io/projected/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-token\") pod \"be263888-3047-4ee6-88cf-0fe7a033f3e1\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") "
Oct 30 22:35:50.804148 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.804082    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-workload-socket\") pod \"be263888-3047-4ee6-88cf-0fe7a033f3e1\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") "
Oct 30 22:35:50.804148 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.804125    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-data\") pod \"be263888-3047-4ee6-88cf-0fe7a033f3e1\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") "
Oct 30 22:35:50.804506 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.804255    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-credential-socket\") pod \"be263888-3047-4ee6-88cf-0fe7a033f3e1\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") "
Oct 30 22:35:50.804506 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.804386    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be263888-3047-4ee6-88cf-0fe7a033f3e1-kube-api-access-z86xp\" (UniqueName: \"kubernetes.io/projected/be263888-3047-4ee6-88cf-0fe7a033f3e1-kube-api-access-z86xp\") pod \"be263888-3047-4ee6-88cf-0fe7a033f3e1\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") "
Oct 30 22:35:50.804506 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.804402    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-workload-certs" pod "be263888-3047-4ee6-88cf-0fe7a033f3e1" (UID: "be263888-3047-4ee6-88cf-0fe7a033f3e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:50.804506 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.804475    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-envoy\") pod \"be263888-3047-4ee6-88cf-0fe7a033f3e1\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") "
Oct 30 22:35:50.804740 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.804518    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be263888-3047-4ee6-88cf-0fe7a033f3e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be263888-3047-4ee6-88cf-0fe7a033f3e1-istiod-ca-cert\") pod \"be263888-3047-4ee6-88cf-0fe7a033f3e1\" (UID: \"be263888-3047-4ee6-88cf-0fe7a033f3e1\") "
Oct 30 22:35:50.804740 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.804524    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-data" pod "be263888-3047-4ee6-88cf-0fe7a033f3e1" (UID: "be263888-3047-4ee6-88cf-0fe7a033f3e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:50.804740 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.804560    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-workload-socket" pod "be263888-3047-4ee6-88cf-0fe7a033f3e1" (UID: "be263888-3047-4ee6-88cf-0fe7a033f3e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:50.804740 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.804666    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-credential-socket" pod "be263888-3047-4ee6-88cf-0fe7a033f3e1" (UID: "be263888-3047-4ee6-88cf-0fe7a033f3e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:50.804972 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.804881    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be263888-3047-4ee6-88cf-0fe7a033f3e1-istiod-ca-cert" pod "be263888-3047-4ee6-88cf-0fe7a033f3e1" (UID: "be263888-3047-4ee6-88cf-0fe7a033f3e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:50.806456 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.805594    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:35:50.806456 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.805620    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:35:50.806456 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.805632    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:35:50.806456 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.805647    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:35:50.806456 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.805659    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be263888-3047-4ee6-88cf-0fe7a033f3e1-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:35:50.807163 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.807128    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-podinfo" pod "be263888-3047-4ee6-88cf-0fe7a033f3e1" (UID: "be263888-3047-4ee6-88cf-0fe7a033f3e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:35:50.807304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.807285    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-token" pod "be263888-3047-4ee6-88cf-0fe7a033f3e1" (UID: "be263888-3047-4ee6-88cf-0fe7a033f3e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:50.807389 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.807374    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be263888-3047-4ee6-88cf-0fe7a033f3e1-kube-api-access-z86xp" pod "be263888-3047-4ee6-88cf-0fe7a033f3e1" (UID: "be263888-3047-4ee6-88cf-0fe7a033f3e1"). InnerVolumeSpecName "kube-api-access-z86xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:50.807445 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.807388    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-envoy" pod "be263888-3047-4ee6-88cf-0fe7a033f3e1" (UID: "be263888-3047-4ee6-88cf-0fe7a033f3e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:50.906930 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.906894    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z86xp\" (UniqueName: \"kubernetes.io/projected/be263888-3047-4ee6-88cf-0fe7a033f3e1-kube-api-access-z86xp\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:35:50.906930 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.906928    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:35:50.906930 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.906938    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:35:50.907174 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:50.906946    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be263888-3047-4ee6-88cf-0fe7a033f3e1-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:35:51.198764 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:51.198675    4889 generic.go:356] "Generic (PLEG): container finished" podID="be263888-3047-4ee6-88cf-0fe7a033f3e1" containerID="a030ba12d8f39b8aceeedc969b6fa04529aad4c194e41ad6ca4ba46b6ac026ec" exitCode=0
Oct 30 22:35:51.198764 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:51.198722    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"
Oct 30 22:35:51.199021 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:51.198768    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp" event={"ID":"be263888-3047-4ee6-88cf-0fe7a033f3e1","Type":"ContainerDied","Data":"a030ba12d8f39b8aceeedc969b6fa04529aad4c194e41ad6ca4ba46b6ac026ec"}
Oct 30 22:35:51.199021 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:51.198827    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp" event={"ID":"be263888-3047-4ee6-88cf-0fe7a033f3e1","Type":"ContainerDied","Data":"aa8bf6f74c3b715269d0cf36679cf9b887726c605f5566794d9769e31d16f68e"}
Oct 30 22:35:51.199021 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:51.198858    4889 scope.go:122] "RemoveContainer" containerID="a030ba12d8f39b8aceeedc969b6fa04529aad4c194e41ad6ca4ba46b6ac026ec"
Oct 30 22:35:51.239423 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:51.239393    4889 scope.go:122] "RemoveContainer" containerID="78ef6659fd1b429d3b13edf3a54e493019dfae74033c9ecb4d421143ba19d1b9"
Oct 30 22:35:51.251595 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:51.251570    4889 scope.go:122] "RemoveContainer" containerID="a030ba12d8f39b8aceeedc969b6fa04529aad4c194e41ad6ca4ba46b6ac026ec"
Oct 30 22:35:51.251914 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:35:51.251891    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a030ba12d8f39b8aceeedc969b6fa04529aad4c194e41ad6ca4ba46b6ac026ec\": container with ID starting with a030ba12d8f39b8aceeedc969b6fa04529aad4c194e41ad6ca4ba46b6ac026ec not found: ID does not exist" containerID="a030ba12d8f39b8aceeedc969b6fa04529aad4c194e41ad6ca4ba46b6ac026ec"
Oct 30 22:35:51.252023 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:51.251923    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a030ba12d8f39b8aceeedc969b6fa04529aad4c194e41ad6ca4ba46b6ac026ec"} err="failed to get container status \"a030ba12d8f39b8aceeedc969b6fa04529aad4c194e41ad6ca4ba46b6ac026ec\": rpc error: code = NotFound desc = could not find container \"a030ba12d8f39b8aceeedc969b6fa04529aad4c194e41ad6ca4ba46b6ac026ec\": container with ID starting with a030ba12d8f39b8aceeedc969b6fa04529aad4c194e41ad6ca4ba46b6ac026ec not found: ID does not exist"
Oct 30 22:35:51.252023 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:51.251940    4889 scope.go:122] "RemoveContainer" containerID="78ef6659fd1b429d3b13edf3a54e493019dfae74033c9ecb4d421143ba19d1b9"
Oct 30 22:35:51.252182 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:35:51.252164    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78ef6659fd1b429d3b13edf3a54e493019dfae74033c9ecb4d421143ba19d1b9\": container with ID starting with 78ef6659fd1b429d3b13edf3a54e493019dfae74033c9ecb4d421143ba19d1b9 not found: ID does not exist" containerID="78ef6659fd1b429d3b13edf3a54e493019dfae74033c9ecb4d421143ba19d1b9"
Oct 30 22:35:51.252279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:51.252182    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78ef6659fd1b429d3b13edf3a54e493019dfae74033c9ecb4d421143ba19d1b9"} err="failed to get container status \"78ef6659fd1b429d3b13edf3a54e493019dfae74033c9ecb4d421143ba19d1b9\": rpc error: code = NotFound desc = could not find container \"78ef6659fd1b429d3b13edf3a54e493019dfae74033c9ecb4d421143ba19d1b9\": container with ID starting with 78ef6659fd1b429d3b13edf3a54e493019dfae74033c9ecb4d421143ba19d1b9 not found: ID does not exist"
Oct 30 22:35:51.279648 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:51.279613    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"]
Oct 30 22:35:51.286329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:51.286301    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-3d38f6-openshift-default-79b69d9955-wjvpp"]
Oct 30 22:35:51.685250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:51.684798    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:52.267268 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.267239    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:52.339319 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.339287    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99"]
Oct 30 22:35:52.340172 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.340149    4889 cpu_manager.go:414] "containerMap: removing container" podUID="be263888-3047-4ee6-88cf-0fe7a033f3e1" containerName="istio-proxy"
Oct 30 22:35:52.340172 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.340169    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be263888-3047-4ee6-88cf-0fe7a033f3e1" containerName="istio-proxy"
Oct 30 22:35:52.340419 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.340181    4889 cpu_manager.go:414] "containerMap: removing container" podUID="be263888-3047-4ee6-88cf-0fe7a033f3e1" containerName="istio-proxy"
Oct 30 22:35:52.340419 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.340186    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be263888-3047-4ee6-88cf-0fe7a033f3e1" containerName="istio-proxy"
Oct 30 22:35:52.340419 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.340355    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="be263888-3047-4ee6-88cf-0fe7a033f3e1" containerName="istio-proxy"
Oct 30 22:35:52.346203 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.346180    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99"
Oct 30 22:35:52.347438 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.347410    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99"]
Oct 30 22:35:52.361851 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.361826    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d85da4\""
Oct 30 22:35:52.386697 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.386669    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp"]
Oct 30 22:35:52.387892 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.387868    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="be263888-3047-4ee6-88cf-0fe7a033f3e1" containerName="istio-proxy"
Oct 30 22:35:52.391758 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.391740    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp"
Oct 30 22:35:52.407176 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.407152    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp"]
Oct 30 22:35:52.471464 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.471425    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d85da4-dockercfg-q4f5h\""
Oct 30 22:35:52.487483 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.481272    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:52.583342 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.582850    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p486\" (UniqueName: \"kubernetes.io/projected/321419c3-5cd7-4e75-8646-b0573c312780-kube-api-access-6p486\") pod \"payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99\" (UID: \"321419c3-5cd7-4e75-8646-b0573c312780\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99"
Oct 30 22:35:52.583342 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.582958    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321419c3-5cd7-4e75-8646-b0573c312780-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99\" (UID: \"321419c3-5cd7-4e75-8646-b0573c312780\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99"
Oct 30 22:35:52.583342 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.583131    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae61647a-51bb-4a9b-b51e-df9559cc9ab2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp\" (UID: \"ae61647a-51bb-4a9b-b51e-df9559cc9ab2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp"
Oct 30 22:35:52.583342 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.583175    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8t8g\" (UniqueName: \"kubernetes.io/projected/ae61647a-51bb-4a9b-b51e-df9559cc9ab2-kube-api-access-m8t8g\") pod \"payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp\" (UID: \"ae61647a-51bb-4a9b-b51e-df9559cc9ab2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp"
Oct 30 22:35:52.723638 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.689357    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321419c3-5cd7-4e75-8646-b0573c312780-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99\" (UID: \"321419c3-5cd7-4e75-8646-b0573c312780\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99"
Oct 30 22:35:52.723638 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.689551    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae61647a-51bb-4a9b-b51e-df9559cc9ab2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp\" (UID: \"ae61647a-51bb-4a9b-b51e-df9559cc9ab2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp"
Oct 30 22:35:52.723638 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.689594    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8t8g\" (UniqueName: \"kubernetes.io/projected/ae61647a-51bb-4a9b-b51e-df9559cc9ab2-kube-api-access-m8t8g\") pod \"payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp\" (UID: \"ae61647a-51bb-4a9b-b51e-df9559cc9ab2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp"
Oct 30 22:35:52.723638 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.689785    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p486\" (UniqueName: \"kubernetes.io/projected/321419c3-5cd7-4e75-8646-b0573c312780-kube-api-access-6p486\") pod \"payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99\" (UID: \"321419c3-5cd7-4e75-8646-b0573c312780\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99"
Oct 30 22:35:52.723638 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.691779    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321419c3-5cd7-4e75-8646-b0573c312780-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99\" (UID: \"321419c3-5cd7-4e75-8646-b0573c312780\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99"
Oct 30 22:35:52.723638 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.694015    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae61647a-51bb-4a9b-b51e-df9559cc9ab2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp\" (UID: \"ae61647a-51bb-4a9b-b51e-df9559cc9ab2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp"
Oct 30 22:35:52.807484 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.795011    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8t8g\" (UniqueName: \"kubernetes.io/projected/ae61647a-51bb-4a9b-b51e-df9559cc9ab2-kube-api-access-m8t8g\") pod \"payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp\" (UID: \"ae61647a-51bb-4a9b-b51e-df9559cc9ab2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp"
Oct 30 22:35:52.807484 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.806898    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p486\" (UniqueName: \"kubernetes.io/projected/321419c3-5cd7-4e75-8646-b0573c312780-kube-api-access-6p486\") pod \"payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99\" (UID: \"321419c3-5cd7-4e75-8646-b0573c312780\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99"
Oct 30 22:35:52.852413 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.852294    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be263888-3047-4ee6-88cf-0fe7a033f3e1" path="/var/lib/kubelet/pods/be263888-3047-4ee6-88cf-0fe7a033f3e1/volumes"
Oct 30 22:35:52.965270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:52.965189    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99"
Oct 30 22:35:53.039900 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:53.007454    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp"
Oct 30 22:35:53.243743 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:53.242035    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:53.441089 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:53.440312    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:53.445311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:53.445278    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp"]
Oct 30 22:35:53.446031 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:35:53.446004    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321419c3_5cd7_4e75_8646_b0573c312780.slice/crio-130b5ddadfa6aff9f362248b4d251e2866835ef6b88f452e13cf969c8b2b1ca6 WatchSource:0}: Error finding container 130b5ddadfa6aff9f362248b4d251e2866835ef6b88f452e13cf969c8b2b1ca6: Status 404 returned error can't find the container with id 130b5ddadfa6aff9f362248b4d251e2866835ef6b88f452e13cf969c8b2b1ca6
Oct 30 22:35:53.447643 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:35:53.447615    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae61647a_51bb_4a9b_b51e_df9559cc9ab2.slice/crio-5c7e32b4f25ffc98c5e00eb644623a1481aa78a0eae80e634b860039a5446846 WatchSource:0}: Error finding container 5c7e32b4f25ffc98c5e00eb644623a1481aa78a0eae80e634b860039a5446846: Status 404 returned error can't find the container with id 5c7e32b4f25ffc98c5e00eb644623a1481aa78a0eae80e634b860039a5446846
Oct 30 22:35:53.450964 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:53.450941    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99"]
Oct 30 22:35:53.772704 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:53.772615    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:54.261834 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:54.261793    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99" event={"ID":"321419c3-5cd7-4e75-8646-b0573c312780","Type":"ContainerStarted","Data":"42d29fbbf650ae812a835c2e28567aea00ba36ed9f3b2d9c4f8165f2780925d7"}
Oct 30 22:35:54.262047 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:54.261850    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99" event={"ID":"321419c3-5cd7-4e75-8646-b0573c312780","Type":"ContainerStarted","Data":"130b5ddadfa6aff9f362248b4d251e2866835ef6b88f452e13cf969c8b2b1ca6"}
Oct 30 22:35:54.262047 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:54.261879    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99"
Oct 30 22:35:54.263689 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:54.263663    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp" event={"ID":"ae61647a-51bb-4a9b-b51e-df9559cc9ab2","Type":"ContainerStarted","Data":"99951b8251a663377daeed11ca306c83691b013e1126504fdbb197365e4021d4"}
Oct 30 22:35:54.263851 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:54.263708    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp" event={"ID":"ae61647a-51bb-4a9b-b51e-df9559cc9ab2","Type":"ContainerStarted","Data":"5c7e32b4f25ffc98c5e00eb644623a1481aa78a0eae80e634b860039a5446846"}
Oct 30 22:35:54.263851 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:54.263849    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp"
Oct 30 22:35:54.302513 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:54.302458    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99" podStartSLOduration=2.302444081 podStartE2EDuration="2.302444081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:54.301149606 +0000 UTC m=+1739.657515913" watchObservedRunningTime="2026-10-30 22:35:54.302444081 +0000 UTC m=+1739.658810393"
Oct 30 22:35:54.355801 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:54.355751    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp" podStartSLOduration=2.355736855 podStartE2EDuration="2.355736855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:54.353984308 +0000 UTC m=+1739.710350624" watchObservedRunningTime="2026-10-30 22:35:54.355736855 +0000 UTC m=+1739.712103170"
Oct 30 22:35:54.749603 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:54.749568    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf" podUID="8646992a-ed0b-4a46-b781-d9e3f5aff39b" containerName="payload-processing" containerID="cri-o://d44919212212e6415030333e16134280e7f917060b235da2a531a681888dbb8b" gracePeriod=25
Oct 30 22:35:54.781481 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:54.781440    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l" podUID="e0a34b6d-354a-4b67-8961-c336db52c10e" containerName="payload-pre-processing" containerID="cri-o://24ba921e10c332fde486d3024e564ea726a49eda70fe03c64f67f088e78f00ca" gracePeriod=25
Oct 30 22:35:54.997421 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:54.996274    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:55.422381 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:55.422289    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"]
Oct 30 22:35:55.422569 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:55.422538    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6" podUID="c2a808e4-e5d8-415e-960e-1b21c1368d1b" containerName="istio-proxy" containerID="cri-o://2ea8f2e33e0c9c0fd3ad0942994dbab0cdb352410231a3f000a05d7cef9d8673" gracePeriod=30
Oct 30 22:35:55.430890 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:55.430864    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"]
Oct 30 22:35:55.443517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:55.441830    4889 generic.go:356] "Generic (PLEG): container finished" podID="8646992a-ed0b-4a46-b781-d9e3f5aff39b" containerID="d44919212212e6415030333e16134280e7f917060b235da2a531a681888dbb8b" exitCode=0
Oct 30 22:35:55.443517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:55.442106    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf" event={"ID":"8646992a-ed0b-4a46-b781-d9e3f5aff39b","Type":"ContainerDied","Data":"d44919212212e6415030333e16134280e7f917060b235da2a531a681888dbb8b"}
Oct 30 22:35:55.451555 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:55.451524    4889 generic.go:356] "Generic (PLEG): container finished" podID="e0a34b6d-354a-4b67-8961-c336db52c10e" containerID="24ba921e10c332fde486d3024e564ea726a49eda70fe03c64f67f088e78f00ca" exitCode=0
Oct 30 22:35:55.453464 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:55.453441    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l" event={"ID":"e0a34b6d-354a-4b67-8961-c336db52c10e","Type":"ContainerDied","Data":"24ba921e10c332fde486d3024e564ea726a49eda70fe03c64f67f088e78f00ca"}
Oct 30 22:35:55.454609 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:55.454581    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c" podUID="794ada71-4cea-4845-90da-72e562678b17" containerName="istio-proxy" containerID="cri-o://4db41ccaaf73d9ab578807990e02b911c3efd9da34d7e6310a5b4611af8023fa" gracePeriod=30
Oct 30 22:35:55.500587 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:55.500544    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf"
Oct 30 22:35:55.505042 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:55.505020    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8646992a-ed0b-4a46-b781-d9e3f5aff39b-kube-api-access-dqrkf\" (UniqueName: \"kubernetes.io/projected/8646992a-ed0b-4a46-b781-d9e3f5aff39b-kube-api-access-dqrkf\") pod \"8646992a-ed0b-4a46-b781-d9e3f5aff39b\" (UID: \"8646992a-ed0b-4a46-b781-d9e3f5aff39b\") "
Oct 30 22:35:55.512293 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:55.512252    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8646992a-ed0b-4a46-b781-d9e3f5aff39b-kube-api-access-dqrkf" pod "8646992a-ed0b-4a46-b781-d9e3f5aff39b" (UID: "8646992a-ed0b-4a46-b781-d9e3f5aff39b"). InnerVolumeSpecName "kube-api-access-dqrkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:55.515922 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:55.515900    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l"
Oct 30 22:35:55.606145 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:55.606115    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8646992a-ed0b-4a46-b781-d9e3f5aff39b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8646992a-ed0b-4a46-b781-d9e3f5aff39b-plugins-config-volume\") pod \"8646992a-ed0b-4a46-b781-d9e3f5aff39b\" (UID: \"8646992a-ed0b-4a46-b781-d9e3f5aff39b\") "
Oct 30 22:35:55.606644 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:55.606447    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0a34b6d-354a-4b67-8961-c336db52c10e-kube-api-access-bkdx8\" (UniqueName: \"kubernetes.io/projected/e0a34b6d-354a-4b67-8961-c336db52c10e-kube-api-access-bkdx8\") pod \"e0a34b6d-354a-4b67-8961-c336db52c10e\" (UID: \"e0a34b6d-354a-4b67-8961-c336db52c10e\") "
Oct 30 22:35:55.606856 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:55.606826    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0a34b6d-354a-4b67-8961-c336db52c10e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a34b6d-354a-4b67-8961-c336db52c10e-plugins-config-volume\") pod \"e0a34b6d-354a-4b67-8961-c336db52c10e\" (UID: \"e0a34b6d-354a-4b67-8961-c336db52c10e\") "
Oct 30 22:35:55.607549 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:55.607529    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqrkf\" (UniqueName: \"kubernetes.io/projected/8646992a-ed0b-4a46-b781-d9e3f5aff39b-kube-api-access-dqrkf\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:35:55.607770 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:55.606516    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8646992a-ed0b-4a46-b781-d9e3f5aff39b-plugins-config-volume" pod "8646992a-ed0b-4a46-b781-d9e3f5aff39b" (UID: "8646992a-ed0b-4a46-b781-d9e3f5aff39b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:55.607897 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:55.607852    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0a34b6d-354a-4b67-8961-c336db52c10e-plugins-config-volume" pod "e0a34b6d-354a-4b67-8961-c336db52c10e" (UID: "e0a34b6d-354a-4b67-8961-c336db52c10e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:55.609491 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:55.609463    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a34b6d-354a-4b67-8961-c336db52c10e-kube-api-access-bkdx8" pod "e0a34b6d-354a-4b67-8961-c336db52c10e" (UID: "e0a34b6d-354a-4b67-8961-c336db52c10e"). InnerVolumeSpecName "kube-api-access-bkdx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:55.709610 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:55.709510    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8646992a-ed0b-4a46-b781-d9e3f5aff39b-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:35:55.709610 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:55.709544    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkdx8\" (UniqueName: \"kubernetes.io/projected/e0a34b6d-354a-4b67-8961-c336db52c10e-kube-api-access-bkdx8\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:35:55.709610 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:55.709557    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a34b6d-354a-4b67-8961-c336db52c10e-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:35:55.948263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:55.944403    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:56.461432 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:56.461400    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l"
Oct 30 22:35:56.461626 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:56.461440    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l" event={"ID":"e0a34b6d-354a-4b67-8961-c336db52c10e","Type":"ContainerDied","Data":"af54f9a911b73d52c5ad379ba049ed6c5098c9cde37d6516ee29894c2845a0a3"}
Oct 30 22:35:56.461626 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:56.461506    4889 scope.go:122] "RemoveContainer" containerID="24ba921e10c332fde486d3024e564ea726a49eda70fe03c64f67f088e78f00ca"
Oct 30 22:35:56.463487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:56.463463    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf"
Oct 30 22:35:56.463487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:56.463471    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf" event={"ID":"8646992a-ed0b-4a46-b781-d9e3f5aff39b","Type":"ContainerDied","Data":"f9bc9716fcca0a808487e12059acf8ffe3c6a24c8ada4e650b49d26e9bd4fc8e"}
Oct 30 22:35:56.474513 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:56.474495    4889 scope.go:122] "RemoveContainer" containerID="d44919212212e6415030333e16134280e7f917060b235da2a531a681888dbb8b"
Oct 30 22:35:56.567938 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:56.567904    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf"]
Oct 30 22:35:56.581248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:56.574111    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-62236a-754c5cbbb9-sxrtf"]
Oct 30 22:35:56.596854 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:56.596807    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l"]
Oct 30 22:35:56.602470 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:56.602427    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-62236a-56cfbd57f9-ldg6l"]
Oct 30 22:35:56.777626 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:56.777593    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8646992a-ed0b-4a46-b781-d9e3f5aff39b" path="/var/lib/kubelet/pods/8646992a-ed0b-4a46-b781-d9e3f5aff39b/volumes"
Oct 30 22:35:56.778238 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:56.778203    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0a34b6d-354a-4b67-8961-c336db52c10e" path="/var/lib/kubelet/pods/e0a34b6d-354a-4b67-8961-c336db52c10e/volumes"
Oct 30 22:35:57.590630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:57.588751    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:58.362807 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:58.362772    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:59.339679 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:35:59.339640    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:00.501921 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.501887    4889 generic.go:356] "Generic (PLEG): container finished" podID="7b49c0bf-aa2d-4f17-8a09-831d9b8bc459" containerID="bf0cb469820dc9eb0b5a8ed43d63125eaa4f4e1a0bd2199e54f55889227bc390" exitCode=0
Oct 30 22:36:00.502519 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.501965    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk" event={"ID":"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459","Type":"ContainerDied","Data":"bf0cb469820dc9eb0b5a8ed43d63125eaa4f4e1a0bd2199e54f55889227bc390"}
Oct 30 22:36:00.502633 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.502615    4889 scope.go:122] "RemoveContainer" containerID="bf0cb469820dc9eb0b5a8ed43d63125eaa4f4e1a0bd2199e54f55889227bc390"
Oct 30 22:36:00.504624 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.504595    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:36:00.504709 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.504670    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:36:00.504709 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.504702    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:36:00.736301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.727204    4889 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 22:36:00.736301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.727326    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c" podUID="794ada71-4cea-4845-90da-72e562678b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 22:36:00.837248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.835313    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:36:00.951235 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.951187    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-envoy\") pod \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") "
Oct 30 22:36:00.951627 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.951601    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-podinfo\") pod \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") "
Oct 30 22:36:00.951778 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.951731    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-token\" (UniqueName: \"kubernetes.io/projected/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-token\") pod \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") "
Oct 30 22:36:00.951878 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.951863    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-workload-certs\") pod \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") "
Oct 30 22:36:00.951946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.951911    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istiod-ca-cert\") pod \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") "
Oct 30 22:36:00.952444 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.951976    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-data\") pod \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") "
Oct 30 22:36:00.952444 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.952123    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-workload-socket\") pod \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") "
Oct 30 22:36:00.952444 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.952192    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-credential-socket\") pod \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") "
Oct 30 22:36:00.952444 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.952284    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2a808e4-e5d8-415e-960e-1b21c1368d1b-kube-api-access-cx7bq\" (UniqueName: \"kubernetes.io/projected/c2a808e4-e5d8-415e-960e-1b21c1368d1b-kube-api-access-cx7bq\") pod \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\" (UID: \"c2a808e4-e5d8-415e-960e-1b21c1368d1b\") "
Oct 30 22:36:00.952716 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.952666    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-workload-socket" pod "c2a808e4-e5d8-415e-960e-1b21c1368d1b" (UID: "c2a808e4-e5d8-415e-960e-1b21c1368d1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:00.952781 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.952762    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-data" pod "c2a808e4-e5d8-415e-960e-1b21c1368d1b" (UID: "c2a808e4-e5d8-415e-960e-1b21c1368d1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:00.952850 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.952790    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-credential-socket" pod "c2a808e4-e5d8-415e-960e-1b21c1368d1b" (UID: "c2a808e4-e5d8-415e-960e-1b21c1368d1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:00.952850 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.952801    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-workload-certs" pod "c2a808e4-e5d8-415e-960e-1b21c1368d1b" (UID: "c2a808e4-e5d8-415e-960e-1b21c1368d1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:00.953168 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.953149    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:00.953168 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.953170    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:00.955691 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.953184    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:00.955691 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.953197    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:00.955691 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.953486    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istiod-ca-cert" pod "c2a808e4-e5d8-415e-960e-1b21c1368d1b" (UID: "c2a808e4-e5d8-415e-960e-1b21c1368d1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:00.955691 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.953663    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:36:00.958032 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.957985    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-envoy" pod "c2a808e4-e5d8-415e-960e-1b21c1368d1b" (UID: "c2a808e4-e5d8-415e-960e-1b21c1368d1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:00.958150 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.958034    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-podinfo" pod "c2a808e4-e5d8-415e-960e-1b21c1368d1b" (UID: "c2a808e4-e5d8-415e-960e-1b21c1368d1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:00.960006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.959981    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a808e4-e5d8-415e-960e-1b21c1368d1b-kube-api-access-cx7bq" pod "c2a808e4-e5d8-415e-960e-1b21c1368d1b" (UID: "c2a808e4-e5d8-415e-960e-1b21c1368d1b"). InnerVolumeSpecName "kube-api-access-cx7bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:00.960265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:00.960204    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-token" pod "c2a808e4-e5d8-415e-960e-1b21c1368d1b" (UID: "c2a808e4-e5d8-415e-960e-1b21c1368d1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:01.053781 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.053744    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/794ada71-4cea-4845-90da-72e562678b17-kube-api-access-lzvj8\" (UniqueName: \"kubernetes.io/projected/794ada71-4cea-4845-90da-72e562678b17-kube-api-access-lzvj8\") pod \"794ada71-4cea-4845-90da-72e562678b17\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") "
Oct 30 22:36:01.053973 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.053799    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-workload-socket\") pod \"794ada71-4cea-4845-90da-72e562678b17\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") "
Oct 30 22:36:01.053973 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.053871    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-workload-certs\") pod \"794ada71-4cea-4845-90da-72e562678b17\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") "
Oct 30 22:36:01.053973 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.053928    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-credential-socket\") pod \"794ada71-4cea-4845-90da-72e562678b17\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") "
Oct 30 22:36:01.054158 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.054100    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/794ada71-4cea-4845-90da-72e562678b17-istio-token\" (UniqueName: \"kubernetes.io/projected/794ada71-4cea-4845-90da-72e562678b17-istio-token\") pod \"794ada71-4cea-4845-90da-72e562678b17\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") "
Oct 30 22:36:01.055487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.054272    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/794ada71-4cea-4845-90da-72e562678b17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/794ada71-4cea-4845-90da-72e562678b17-istio-podinfo\") pod \"794ada71-4cea-4845-90da-72e562678b17\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") "
Oct 30 22:36:01.055487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.054374    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-istio-envoy\") pod \"794ada71-4cea-4845-90da-72e562678b17\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") "
Oct 30 22:36:01.055487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.054454    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/794ada71-4cea-4845-90da-72e562678b17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/794ada71-4cea-4845-90da-72e562678b17-istiod-ca-cert\") pod \"794ada71-4cea-4845-90da-72e562678b17\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") "
Oct 30 22:36:01.055487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.054485    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-istio-data\") pod \"794ada71-4cea-4845-90da-72e562678b17\" (UID: \"794ada71-4cea-4845-90da-72e562678b17\") "
Oct 30 22:36:01.055487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.054543    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-workload-certs" pod "794ada71-4cea-4845-90da-72e562678b17" (UID: "794ada71-4cea-4845-90da-72e562678b17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:01.055487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.054595    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-credential-socket" pod "794ada71-4cea-4845-90da-72e562678b17" (UID: "794ada71-4cea-4845-90da-72e562678b17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:01.055487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.055167    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/794ada71-4cea-4845-90da-72e562678b17-istiod-ca-cert" pod "794ada71-4cea-4845-90da-72e562678b17" (UID: "794ada71-4cea-4845-90da-72e562678b17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:01.055487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.055269    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx7bq\" (UniqueName: \"kubernetes.io/projected/c2a808e4-e5d8-415e-960e-1b21c1368d1b-kube-api-access-cx7bq\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:01.055487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.055292    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:01.055487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.055304    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:01.055487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.055315    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:01.055487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.055328    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a808e4-e5d8-415e-960e-1b21c1368d1b-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:01.055487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.055342    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:01.055487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.055358    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:01.055487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.055357    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-istio-data" pod "794ada71-4cea-4845-90da-72e562678b17" (UID: "794ada71-4cea-4845-90da-72e562678b17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:01.055487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.055427    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-workload-socket" pod "794ada71-4cea-4845-90da-72e562678b17" (UID: "794ada71-4cea-4845-90da-72e562678b17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:01.057656 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.057619    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/794ada71-4cea-4845-90da-72e562678b17-istio-podinfo" pod "794ada71-4cea-4845-90da-72e562678b17" (UID: "794ada71-4cea-4845-90da-72e562678b17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:01.057999 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.057967    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794ada71-4cea-4845-90da-72e562678b17-istio-token" pod "794ada71-4cea-4845-90da-72e562678b17" (UID: "794ada71-4cea-4845-90da-72e562678b17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:01.058751 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.058723    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794ada71-4cea-4845-90da-72e562678b17-kube-api-access-lzvj8" pod "794ada71-4cea-4845-90da-72e562678b17" (UID: "794ada71-4cea-4845-90da-72e562678b17"). InnerVolumeSpecName "kube-api-access-lzvj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:01.058854 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.058836    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-istio-envoy" pod "794ada71-4cea-4845-90da-72e562678b17" (UID: "794ada71-4cea-4845-90da-72e562678b17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:01.157199 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.157158    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/794ada71-4cea-4845-90da-72e562678b17-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:01.157199 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.157193    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:01.157199 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.157203    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/794ada71-4cea-4845-90da-72e562678b17-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:01.157199 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.157210    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:01.157510 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.157241    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzvj8\" (UniqueName: \"kubernetes.io/projected/794ada71-4cea-4845-90da-72e562678b17-kube-api-access-lzvj8\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:01.157510 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.157252    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/794ada71-4cea-4845-90da-72e562678b17-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:01.157510 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.157260    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/794ada71-4cea-4845-90da-72e562678b17-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:01.225910 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.225843    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:01.536509 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.536470    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk" event={"ID":"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459","Type":"ContainerStarted","Data":"4429583f8ee29228a72cea60ac51ff8eafcccbf17f65a9b42882aca907b44044"}
Oct 30 22:36:01.538126 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.538097    4889 generic.go:356] "Generic (PLEG): container finished" podID="c2a808e4-e5d8-415e-960e-1b21c1368d1b" containerID="2ea8f2e33e0c9c0fd3ad0942994dbab0cdb352410231a3f000a05d7cef9d8673" exitCode=0
Oct 30 22:36:01.538274 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.538146    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"
Oct 30 22:36:01.538274 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.538181    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6" event={"ID":"c2a808e4-e5d8-415e-960e-1b21c1368d1b","Type":"ContainerDied","Data":"2ea8f2e33e0c9c0fd3ad0942994dbab0cdb352410231a3f000a05d7cef9d8673"}
Oct 30 22:36:01.538274 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.538249    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6" event={"ID":"c2a808e4-e5d8-415e-960e-1b21c1368d1b","Type":"ContainerDied","Data":"83627d818a1c83f74eb28e7eaf22ceb24a6ee9de0cea1a0c56c28eea6dc08195"}
Oct 30 22:36:01.538430 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.538276    4889 scope.go:122] "RemoveContainer" containerID="2ea8f2e33e0c9c0fd3ad0942994dbab0cdb352410231a3f000a05d7cef9d8673"
Oct 30 22:36:01.539832 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.539812    4889 generic.go:356] "Generic (PLEG): container finished" podID="794ada71-4cea-4845-90da-72e562678b17" containerID="4db41ccaaf73d9ab578807990e02b911c3efd9da34d7e6310a5b4611af8023fa" exitCode=0
Oct 30 22:36:01.539942 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.539858    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"
Oct 30 22:36:01.540009 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.539938    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c" event={"ID":"794ada71-4cea-4845-90da-72e562678b17","Type":"ContainerDied","Data":"4db41ccaaf73d9ab578807990e02b911c3efd9da34d7e6310a5b4611af8023fa"}
Oct 30 22:36:01.540009 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.540002    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c" event={"ID":"794ada71-4cea-4845-90da-72e562678b17","Type":"ContainerDied","Data":"1814d6a75cec6da2b206cd8834f7e79016af08a92edb6160615291e348f8c2ec"}
Oct 30 22:36:01.549898 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.549875    4889 scope.go:122] "RemoveContainer" containerID="2ea8f2e33e0c9c0fd3ad0942994dbab0cdb352410231a3f000a05d7cef9d8673"
Oct 30 22:36:01.550211 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:36:01.550192    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea8f2e33e0c9c0fd3ad0942994dbab0cdb352410231a3f000a05d7cef9d8673\": container with ID starting with 2ea8f2e33e0c9c0fd3ad0942994dbab0cdb352410231a3f000a05d7cef9d8673 not found: ID does not exist" containerID="2ea8f2e33e0c9c0fd3ad0942994dbab0cdb352410231a3f000a05d7cef9d8673"
Oct 30 22:36:01.550308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.550235    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea8f2e33e0c9c0fd3ad0942994dbab0cdb352410231a3f000a05d7cef9d8673"} err="failed to get container status \"2ea8f2e33e0c9c0fd3ad0942994dbab0cdb352410231a3f000a05d7cef9d8673\": rpc error: code = NotFound desc = could not find container \"2ea8f2e33e0c9c0fd3ad0942994dbab0cdb352410231a3f000a05d7cef9d8673\": container with ID starting with 2ea8f2e33e0c9c0fd3ad0942994dbab0cdb352410231a3f000a05d7cef9d8673 not found: ID does not exist"
Oct 30 22:36:01.550308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.550274    4889 scope.go:122] "RemoveContainer" containerID="4db41ccaaf73d9ab578807990e02b911c3efd9da34d7e6310a5b4611af8023fa"
Oct 30 22:36:01.561288 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.561267    4889 scope.go:122] "RemoveContainer" containerID="4db41ccaaf73d9ab578807990e02b911c3efd9da34d7e6310a5b4611af8023fa"
Oct 30 22:36:01.561620 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:36:01.561601    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4db41ccaaf73d9ab578807990e02b911c3efd9da34d7e6310a5b4611af8023fa\": container with ID starting with 4db41ccaaf73d9ab578807990e02b911c3efd9da34d7e6310a5b4611af8023fa not found: ID does not exist" containerID="4db41ccaaf73d9ab578807990e02b911c3efd9da34d7e6310a5b4611af8023fa"
Oct 30 22:36:01.561670 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.561631    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4db41ccaaf73d9ab578807990e02b911c3efd9da34d7e6310a5b4611af8023fa"} err="failed to get container status \"4db41ccaaf73d9ab578807990e02b911c3efd9da34d7e6310a5b4611af8023fa\": rpc error: code = NotFound desc = could not find container \"4db41ccaaf73d9ab578807990e02b911c3efd9da34d7e6310a5b4611af8023fa\": container with ID starting with 4db41ccaaf73d9ab578807990e02b911c3efd9da34d7e6310a5b4611af8023fa not found: ID does not exist"
Oct 30 22:36:01.650448 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.650405    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"]
Oct 30 22:36:01.654697 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.654675    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hnqx6"]
Oct 30 22:36:01.742721 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.742679    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"]
Oct 30 22:36:01.750435 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:01.750399    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-25w9c"]
Oct 30 22:36:02.360636 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:02.360599    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:36:02.361642 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:02.361615    4889 patch_prober.go:29] interesting pod/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 22:36:02.361768 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:02.361663    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk" podUID="7b49c0bf-aa2d-4f17-8a09-831d9b8bc459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 22:36:02.755848 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:02.755807    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:02.776950 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:02.776917    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="794ada71-4cea-4845-90da-72e562678b17" path="/var/lib/kubelet/pods/794ada71-4cea-4845-90da-72e562678b17/volumes"
Oct 30 22:36:02.777513 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:02.777498    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2a808e4-e5d8-415e-960e-1b21c1368d1b" path="/var/lib/kubelet/pods/c2a808e4-e5d8-415e-960e-1b21c1368d1b/volumes"
Oct 30 22:36:03.394247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:03.393597    4889 patch_prober.go:29] interesting pod/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 22:36:03.394247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:03.393667    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk" podUID="7b49c0bf-aa2d-4f17-8a09-831d9b8bc459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 22:36:03.462771 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:03.462411    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:03.649308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:03.644261    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:03.845250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:03.844413    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:04.131250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:04.122868    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:04.361782 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:04.361748    4889 patch_prober.go:29] interesting pod/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 22:36:04.361980 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:04.361804    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk" podUID="7b49c0bf-aa2d-4f17-8a09-831d9b8bc459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 22:36:04.372872 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:04.372844    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:36:05.361540 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:05.361449    4889 patch_prober.go:29] interesting pod/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 22:36:05.361540 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:05.361513    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk" podUID="7b49c0bf-aa2d-4f17-8a09-831d9b8bc459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 22:36:05.456681 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:05.456646    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99"
Oct 30 22:36:05.457968 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:05.457944    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp"
Oct 30 22:36:05.803426 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:05.803392    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:05.998249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:05.992820    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:06.173790 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:06.173686    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:06.361529 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:06.361493    4889 patch_prober.go:29] interesting pod/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 22:36:06.361710 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:06.361549    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk" podUID="7b49c0bf-aa2d-4f17-8a09-831d9b8bc459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 22:36:07.361674 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:07.361635    4889 patch_prober.go:29] interesting pod/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 22:36:07.361873 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:07.361717    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk" podUID="7b49c0bf-aa2d-4f17-8a09-831d9b8bc459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 22:36:08.022707 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:08.022671    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:08.366981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:08.366904    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:36:08.368178 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:08.368161    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:36:09.476138 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.476106    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h"]
Oct 30 22:36:09.476915 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.476900    4889 cpu_manager.go:414] "containerMap: removing container" podUID="e0a34b6d-354a-4b67-8961-c336db52c10e" containerName="payload-pre-processing"
Oct 30 22:36:09.476962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.476917    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0a34b6d-354a-4b67-8961-c336db52c10e" containerName="payload-pre-processing"
Oct 30 22:36:09.476962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.476935    4889 cpu_manager.go:414] "containerMap: removing container" podUID="8646992a-ed0b-4a46-b781-d9e3f5aff39b" containerName="payload-processing"
Oct 30 22:36:09.476962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.476941    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8646992a-ed0b-4a46-b781-d9e3f5aff39b" containerName="payload-processing"
Oct 30 22:36:09.476962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.476951    4889 cpu_manager.go:414] "containerMap: removing container" podUID="794ada71-4cea-4845-90da-72e562678b17" containerName="istio-proxy"
Oct 30 22:36:09.476962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.476957    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="794ada71-4cea-4845-90da-72e562678b17" containerName="istio-proxy"
Oct 30 22:36:09.477124 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.476988    4889 cpu_manager.go:414] "containerMap: removing container" podUID="c2a808e4-e5d8-415e-960e-1b21c1368d1b" containerName="istio-proxy"
Oct 30 22:36:09.477124 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.476992    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2a808e4-e5d8-415e-960e-1b21c1368d1b" containerName="istio-proxy"
Oct 30 22:36:09.477189 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.477122    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0a34b6d-354a-4b67-8961-c336db52c10e" containerName="payload-pre-processing"
Oct 30 22:36:09.477189 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.477135    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="8646992a-ed0b-4a46-b781-d9e3f5aff39b" containerName="payload-processing"
Oct 30 22:36:09.477189 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.477144    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="794ada71-4cea-4845-90da-72e562678b17" containerName="istio-proxy"
Oct 30 22:36:09.477189 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.477154    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2a808e4-e5d8-415e-960e-1b21c1368d1b" containerName="istio-proxy"
Oct 30 22:36:09.482480 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.482463    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h"
Oct 30 22:36:09.504509 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.504478    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h"]
Oct 30 22:36:09.510312 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.510270    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-6013c8-serving-cert\""
Oct 30 22:36:09.595782 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.595746    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2241313-b99d-4804-9c2e-eb54319c1668-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h\" (UID: \"d2241313-b99d-4804-9c2e-eb54319c1668\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h"
Oct 30 22:36:09.595995 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.595799    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2241313-b99d-4804-9c2e-eb54319c1668-metrics-certs\") pod \"maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h\" (UID: \"d2241313-b99d-4804-9c2e-eb54319c1668\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h"
Oct 30 22:36:09.595995 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.595906    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vls9f\" (UniqueName: \"kubernetes.io/projected/d2241313-b99d-4804-9c2e-eb54319c1668-kube-api-access-vls9f\") pod \"maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h\" (UID: \"d2241313-b99d-4804-9c2e-eb54319c1668\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h"
Oct 30 22:36:09.596130 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.596107    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2241313-b99d-4804-9c2e-eb54319c1668-maas-api-tls\") pod \"maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h\" (UID: \"d2241313-b99d-4804-9c2e-eb54319c1668\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h"
Oct 30 22:36:09.626281 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.626243    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:09.710258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.703680    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2241313-b99d-4804-9c2e-eb54319c1668-maas-api-tls\") pod \"maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h\" (UID: \"d2241313-b99d-4804-9c2e-eb54319c1668\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h"
Oct 30 22:36:09.716390 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.716349    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2241313-b99d-4804-9c2e-eb54319c1668-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h\" (UID: \"d2241313-b99d-4804-9c2e-eb54319c1668\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h"
Oct 30 22:36:09.727635 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.727546    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2241313-b99d-4804-9c2e-eb54319c1668-metrics-certs\") pod \"maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h\" (UID: \"d2241313-b99d-4804-9c2e-eb54319c1668\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h"
Oct 30 22:36:09.728464 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.728439    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vls9f\" (UniqueName: \"kubernetes.io/projected/d2241313-b99d-4804-9c2e-eb54319c1668-kube-api-access-vls9f\") pod \"maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h\" (UID: \"d2241313-b99d-4804-9c2e-eb54319c1668\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h"
Oct 30 22:36:09.738557 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.719751    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2241313-b99d-4804-9c2e-eb54319c1668-maas-api-tls\") pod \"maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h\" (UID: \"d2241313-b99d-4804-9c2e-eb54319c1668\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h"
Oct 30 22:36:09.738873 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.720702    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2241313-b99d-4804-9c2e-eb54319c1668-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h\" (UID: \"d2241313-b99d-4804-9c2e-eb54319c1668\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h"
Oct 30 22:36:09.744456 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.744420    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2241313-b99d-4804-9c2e-eb54319c1668-metrics-certs\") pod \"maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h\" (UID: \"d2241313-b99d-4804-9c2e-eb54319c1668\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h"
Oct 30 22:36:09.767396 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.760810    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vls9f\" (UniqueName: \"kubernetes.io/projected/d2241313-b99d-4804-9c2e-eb54319c1668-kube-api-access-vls9f\") pod \"maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h\" (UID: \"d2241313-b99d-4804-9c2e-eb54319c1668\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h"
Oct 30 22:36:09.805302 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.803567    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h"
Oct 30 22:36:09.807698 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:09.806305    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:10.129751 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:36:10.129717    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2241313_b99d_4804_9c2e_eb54319c1668.slice/crio-289706a67152eedbbb479b0037f114fbc6d0a51444ec3333d9a1dc7c8dd2c8c1 WatchSource:0}: Error finding container 289706a67152eedbbb479b0037f114fbc6d0a51444ec3333d9a1dc7c8dd2c8c1: Status 404 returned error can't find the container with id 289706a67152eedbbb479b0037f114fbc6d0a51444ec3333d9a1dc7c8dd2c8c1
Oct 30 22:36:10.130865 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:10.130841    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h"]
Oct 30 22:36:10.303101 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:36:10.303067    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:36:10.664770 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:10.664725    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h" event={"ID":"d2241313-b99d-4804-9c2e-eb54319c1668","Type":"ContainerStarted","Data":"945b7df290f39c5bc7474c4b7dd023593f26ef821a30ae091da6f6d62d07f8b5"}
Oct 30 22:36:10.665346 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:10.664800    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h" event={"ID":"d2241313-b99d-4804-9c2e-eb54319c1668","Type":"ContainerStarted","Data":"289706a67152eedbbb479b0037f114fbc6d0a51444ec3333d9a1dc7c8dd2c8c1"}
Oct 30 22:36:10.665346 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:10.664829    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h"
Oct 30 22:36:10.784341 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:10.717116    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h" podStartSLOduration=1.388596371 podStartE2EDuration="1.717094175s" totalImagesPullingTime="328.497804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:10.708711591 +0000 UTC m=+1756.065077930" watchObservedRunningTime="2026-10-30 22:36:10.717094175 +0000 UTC m=+1756.073460491"
Oct 30 22:36:10.902602 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:10.902567    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:11.080313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:11.079872    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:11.177294 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:11.177213    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"]
Oct 30 22:36:11.177507 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:11.177482    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p" podUID="47a318ba-6fbd-479f-aa9a-e07dc2ab12a8" containerName="istio-proxy" containerID="cri-o://01918c94a8f5ead28c67206e949821698055c2d2b0be5203dc712757e723491a" gracePeriod=30
Oct 30 22:36:11.296251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:11.291858    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:11.744254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:11.734677    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:12.063068 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:12.062978    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:12.517592 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:12.517529    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:12.694254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:12.694194    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:13.504716 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:13.504672    4889 patch_prober.go:29] interesting pod/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:36:13.505182 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:13.504761    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p" podUID="47a318ba-6fbd-479f-aa9a-e07dc2ab12a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:36:13.743132 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:13.743079    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:15.526248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:15.520875    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:16.471830 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.471805    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:36:16.548185 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.546915    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-podinfo\") pod \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") "
Oct 30 22:36:16.548185 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.547014    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-credential-socket\") pod \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") "
Oct 30 22:36:16.548185 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.547076    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-kube-api-access-cplvh\" (UniqueName: \"kubernetes.io/projected/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-kube-api-access-cplvh\") pod \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") "
Oct 30 22:36:16.548185 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.547114    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-workload-socket\") pod \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") "
Oct 30 22:36:16.548185 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.547215    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-workload-certs\") pod \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") "
Oct 30 22:36:16.548185 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.547271    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-data\") pod \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") "
Oct 30 22:36:16.548185 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.547305    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istiod-ca-cert\") pod \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") "
Oct 30 22:36:16.548185 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.547342    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-token\" (UniqueName: \"kubernetes.io/projected/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-token\") pod \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") "
Oct 30 22:36:16.548185 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.547405    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-envoy\") pod \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\" (UID: \"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8\") "
Oct 30 22:36:16.548185 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.547598    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-workload-socket" pod "47a318ba-6fbd-479f-aa9a-e07dc2ab12a8" (UID: "47a318ba-6fbd-479f-aa9a-e07dc2ab12a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:16.548185 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.547952    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-workload-certs" pod "47a318ba-6fbd-479f-aa9a-e07dc2ab12a8" (UID: "47a318ba-6fbd-479f-aa9a-e07dc2ab12a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:16.548185 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.548192    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-data" pod "47a318ba-6fbd-479f-aa9a-e07dc2ab12a8" (UID: "47a318ba-6fbd-479f-aa9a-e07dc2ab12a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:16.549406 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.548336    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:16.549406 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.548373    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:16.549406 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.548371    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istiod-ca-cert" pod "47a318ba-6fbd-479f-aa9a-e07dc2ab12a8" (UID: "47a318ba-6fbd-479f-aa9a-e07dc2ab12a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:16.549406 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.548432    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-credential-socket" pod "47a318ba-6fbd-479f-aa9a-e07dc2ab12a8" (UID: "47a318ba-6fbd-479f-aa9a-e07dc2ab12a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:16.551601 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.551574    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-token" pod "47a318ba-6fbd-479f-aa9a-e07dc2ab12a8" (UID: "47a318ba-6fbd-479f-aa9a-e07dc2ab12a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:16.552570 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.552538    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-envoy" pod "47a318ba-6fbd-479f-aa9a-e07dc2ab12a8" (UID: "47a318ba-6fbd-479f-aa9a-e07dc2ab12a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:16.555678 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.555655    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-podinfo" pod "47a318ba-6fbd-479f-aa9a-e07dc2ab12a8" (UID: "47a318ba-6fbd-479f-aa9a-e07dc2ab12a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:16.555678 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.555668    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-kube-api-access-cplvh" pod "47a318ba-6fbd-479f-aa9a-e07dc2ab12a8" (UID: "47a318ba-6fbd-479f-aa9a-e07dc2ab12a8"). InnerVolumeSpecName "kube-api-access-cplvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:16.618269 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.618210    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:16.650469 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.650442    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:16.650469 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.650468    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:16.650667 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.650480    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cplvh\" (UniqueName: \"kubernetes.io/projected/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-kube-api-access-cplvh\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:16.650667 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.650491    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:16.650667 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.650500    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:16.650667 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.650507    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:16.650667 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.650514    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:16.680774 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.680737    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h"
Oct 30 22:36:16.790920 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.790542    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:16.816091 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.816057    4889 generic.go:356] "Generic (PLEG): container finished" podID="47a318ba-6fbd-479f-aa9a-e07dc2ab12a8" containerID="01918c94a8f5ead28c67206e949821698055c2d2b0be5203dc712757e723491a" exitCode=0
Oct 30 22:36:16.816275 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.816113    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"
Oct 30 22:36:16.816275 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.816134    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p" event={"ID":"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8","Type":"ContainerDied","Data":"01918c94a8f5ead28c67206e949821698055c2d2b0be5203dc712757e723491a"}
Oct 30 22:36:16.816275 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.816212    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p" event={"ID":"47a318ba-6fbd-479f-aa9a-e07dc2ab12a8","Type":"ContainerDied","Data":"e05826b9bce92a15a7abb6ecf1220018087733c9b390b8e714c4e96866b355ce"}
Oct 30 22:36:16.816448 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.816293    4889 scope.go:122] "RemoveContainer" containerID="01918c94a8f5ead28c67206e949821698055c2d2b0be5203dc712757e723491a"
Oct 30 22:36:16.828451 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.828432    4889 scope.go:122] "RemoveContainer" containerID="d73d742efb808e8ea43f9a3b9ae9c0e1a94061777bdd868b121bc998e8c61570"
Oct 30 22:36:16.858422 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.858395    4889 scope.go:122] "RemoveContainer" containerID="01918c94a8f5ead28c67206e949821698055c2d2b0be5203dc712757e723491a"
Oct 30 22:36:16.858854 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:36:16.858829    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01918c94a8f5ead28c67206e949821698055c2d2b0be5203dc712757e723491a\": container with ID starting with 01918c94a8f5ead28c67206e949821698055c2d2b0be5203dc712757e723491a not found: ID does not exist" containerID="01918c94a8f5ead28c67206e949821698055c2d2b0be5203dc712757e723491a"
Oct 30 22:36:16.858975 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.858866    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01918c94a8f5ead28c67206e949821698055c2d2b0be5203dc712757e723491a"} err="failed to get container status \"01918c94a8f5ead28c67206e949821698055c2d2b0be5203dc712757e723491a\": rpc error: code = NotFound desc = could not find container \"01918c94a8f5ead28c67206e949821698055c2d2b0be5203dc712757e723491a\": container with ID starting with 01918c94a8f5ead28c67206e949821698055c2d2b0be5203dc712757e723491a not found: ID does not exist"
Oct 30 22:36:16.858975 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.858893    4889 scope.go:122] "RemoveContainer" containerID="d73d742efb808e8ea43f9a3b9ae9c0e1a94061777bdd868b121bc998e8c61570"
Oct 30 22:36:16.859292 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:36:16.859270    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d73d742efb808e8ea43f9a3b9ae9c0e1a94061777bdd868b121bc998e8c61570\": container with ID starting with d73d742efb808e8ea43f9a3b9ae9c0e1a94061777bdd868b121bc998e8c61570 not found: ID does not exist" containerID="d73d742efb808e8ea43f9a3b9ae9c0e1a94061777bdd868b121bc998e8c61570"
Oct 30 22:36:16.859366 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.859300    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d73d742efb808e8ea43f9a3b9ae9c0e1a94061777bdd868b121bc998e8c61570"} err="failed to get container status \"d73d742efb808e8ea43f9a3b9ae9c0e1a94061777bdd868b121bc998e8c61570\": rpc error: code = NotFound desc = could not find container \"d73d742efb808e8ea43f9a3b9ae9c0e1a94061777bdd868b121bc998e8c61570\": container with ID starting with d73d742efb808e8ea43f9a3b9ae9c0e1a94061777bdd868b121bc998e8c61570 not found: ID does not exist"
Oct 30 22:36:16.873604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.873583    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"]
Oct 30 22:36:16.879908 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:16.879888    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-8b3e43-openshift-default-78bccfc7fc-n448p"]
Oct 30 22:36:17.570908 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:17.570859    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:18.316352 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:18.316321    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:18.379013 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:18.378979    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-40229e-86586c78fc-288bj"]
Oct 30 22:36:18.379856 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:18.379837    4889 cpu_manager.go:414] "containerMap: removing container" podUID="47a318ba-6fbd-479f-aa9a-e07dc2ab12a8" containerName="istio-proxy"
Oct 30 22:36:18.379905 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:18.379859    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47a318ba-6fbd-479f-aa9a-e07dc2ab12a8" containerName="istio-proxy"
Oct 30 22:36:18.379905 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:18.379877    4889 cpu_manager.go:414] "containerMap: removing container" podUID="47a318ba-6fbd-479f-aa9a-e07dc2ab12a8" containerName="istio-proxy"
Oct 30 22:36:18.379905 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:18.379883    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47a318ba-6fbd-479f-aa9a-e07dc2ab12a8" containerName="istio-proxy"
Oct 30 22:36:18.380035 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:18.380026    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="47a318ba-6fbd-479f-aa9a-e07dc2ab12a8" containerName="istio-proxy"
Oct 30 22:36:18.380076 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:18.380037    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="47a318ba-6fbd-479f-aa9a-e07dc2ab12a8" containerName="istio-proxy"
Oct 30 22:36:18.385132 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:18.385112    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-40229e-86586c78fc-288bj"
Oct 30 22:36:18.401634 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:18.401609    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-40229e-86586c78fc-288bj"]
Oct 30 22:36:18.409702 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:18.409668    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-40229e\""
Oct 30 22:36:18.409857 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:18.409712    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-40229e-dockercfg-2m86v\""
Oct 30 22:36:18.515259 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:18.507375    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbfwm\" (UniqueName: \"kubernetes.io/projected/3b894efe-e301-4948-a86c-b04c0450f7d0-kube-api-access-lbfwm\") pod \"payload-processing-e2e-inf-a-40229e-86586c78fc-288bj\" (UID: \"3b894efe-e301-4948-a86c-b04c0450f7d0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-40229e-86586c78fc-288bj"
Oct 30 22:36:18.515259 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:18.507469    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b894efe-e301-4948-a86c-b04c0450f7d0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-40229e-86586c78fc-288bj\" (UID: \"3b894efe-e301-4948-a86c-b04c0450f7d0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-40229e-86586c78fc-288bj"
Oct 30 22:36:18.557252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:18.551385    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:18.627332 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:18.619209    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbfwm\" (UniqueName: \"kubernetes.io/projected/3b894efe-e301-4948-a86c-b04c0450f7d0-kube-api-access-lbfwm\") pod \"payload-processing-e2e-inf-a-40229e-86586c78fc-288bj\" (UID: \"3b894efe-e301-4948-a86c-b04c0450f7d0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-40229e-86586c78fc-288bj"
Oct 30 22:36:18.627332 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:18.619310    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b894efe-e301-4948-a86c-b04c0450f7d0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-40229e-86586c78fc-288bj\" (UID: \"3b894efe-e301-4948-a86c-b04c0450f7d0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-40229e-86586c78fc-288bj"
Oct 30 22:36:18.627332 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:18.620603    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b894efe-e301-4948-a86c-b04c0450f7d0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-40229e-86586c78fc-288bj\" (UID: \"3b894efe-e301-4948-a86c-b04c0450f7d0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-40229e-86586c78fc-288bj"
Oct 30 22:36:18.669125 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:18.668245    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbfwm\" (UniqueName: \"kubernetes.io/projected/3b894efe-e301-4948-a86c-b04c0450f7d0-kube-api-access-lbfwm\") pod \"payload-processing-e2e-inf-a-40229e-86586c78fc-288bj\" (UID: \"3b894efe-e301-4948-a86c-b04c0450f7d0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-40229e-86586c78fc-288bj"
Oct 30 22:36:18.704910 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:18.704745    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-40229e-86586c78fc-288bj"
Oct 30 22:36:18.789819 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:18.788608    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47a318ba-6fbd-479f-aa9a-e07dc2ab12a8" path="/var/lib/kubelet/pods/47a318ba-6fbd-479f-aa9a-e07dc2ab12a8/volumes"
Oct 30 22:36:19.089706 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:36:19.089664    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b894efe_e301_4948_a86c_b04c0450f7d0.slice/crio-c6bd77973f04a0c7c9f6158e4df60a374968a68c6126d4238752af226e13812c WatchSource:0}: Error finding container c6bd77973f04a0c7c9f6158e4df60a374968a68c6126d4238752af226e13812c: Status 404 returned error can't find the container with id c6bd77973f04a0c7c9f6158e4df60a374968a68c6126d4238752af226e13812c
Oct 30 22:36:19.090711 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:19.090688    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-40229e-86586c78fc-288bj"]
Oct 30 22:36:19.768943 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:19.768898    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:19.846529 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:19.846496    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-40229e-86586c78fc-288bj" event={"ID":"3b894efe-e301-4948-a86c-b04c0450f7d0","Type":"ContainerStarted","Data":"f7a540ec0c81cfb82ac51385aded187d6c600e97e9ac3d1a04f13672c4fe182a"}
Oct 30 22:36:19.846700 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:19.846555    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-40229e-86586c78fc-288bj" event={"ID":"3b894efe-e301-4948-a86c-b04c0450f7d0","Type":"ContainerStarted","Data":"c6bd77973f04a0c7c9f6158e4df60a374968a68c6126d4238752af226e13812c"}
Oct 30 22:36:19.846785 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:19.846709    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-40229e-86586c78fc-288bj"
Oct 30 22:36:19.886856 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:19.886801    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-40229e-86586c78fc-288bj" podStartSLOduration=1.886787296 podStartE2EDuration="1.886787296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:19.885192675 +0000 UTC m=+1765.241558992" watchObservedRunningTime="2026-10-30 22:36:19.886787296 +0000 UTC m=+1765.243153610"
Oct 30 22:36:21.559405 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:21.559358    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:23.409631 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:23.409583    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:25.094683 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:25.094641    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:25.309257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:25.304344    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:27.118355 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:27.118318    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:29.271251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:29.269735    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:30.857914 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:30.857879    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-40229e-86586c78fc-288bj"
Oct 30 22:36:30.947459 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:30.947423    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8"]
Oct 30 22:36:30.963021 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:30.962994    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8"]
Oct 30 22:36:30.963277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:30.963106    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8"
Oct 30 22:36:30.987044 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:30.981799    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d85da4-serving-cert\""
Oct 30 22:36:30.987044 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:30.986989    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c660853-ad95-4bc2-b1f5-52690a8e728f-metrics-certs\") pod \"maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8\" (UID: \"1c660853-ad95-4bc2-b1f5-52690a8e728f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8"
Oct 30 22:36:30.987291 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:30.987247    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw9nx\" (UniqueName: \"kubernetes.io/projected/1c660853-ad95-4bc2-b1f5-52690a8e728f-kube-api-access-gw9nx\") pod \"maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8\" (UID: \"1c660853-ad95-4bc2-b1f5-52690a8e728f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8"
Oct 30 22:36:30.987505 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:30.987481    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c660853-ad95-4bc2-b1f5-52690a8e728f-maas-api-tls\") pod \"maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8\" (UID: \"1c660853-ad95-4bc2-b1f5-52690a8e728f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8"
Oct 30 22:36:30.987650 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:30.987629    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c660853-ad95-4bc2-b1f5-52690a8e728f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8\" (UID: \"1c660853-ad95-4bc2-b1f5-52690a8e728f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8"
Oct 30 22:36:31.089589 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:31.089558    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c660853-ad95-4bc2-b1f5-52690a8e728f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8\" (UID: \"1c660853-ad95-4bc2-b1f5-52690a8e728f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8"
Oct 30 22:36:31.089802 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:31.089678    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c660853-ad95-4bc2-b1f5-52690a8e728f-metrics-certs\") pod \"maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8\" (UID: \"1c660853-ad95-4bc2-b1f5-52690a8e728f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8"
Oct 30 22:36:31.089862 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:31.089839    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw9nx\" (UniqueName: \"kubernetes.io/projected/1c660853-ad95-4bc2-b1f5-52690a8e728f-kube-api-access-gw9nx\") pod \"maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8\" (UID: \"1c660853-ad95-4bc2-b1f5-52690a8e728f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8"
Oct 30 22:36:31.089900 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:31.089869    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c660853-ad95-4bc2-b1f5-52690a8e728f-maas-api-tls\") pod \"maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8\" (UID: \"1c660853-ad95-4bc2-b1f5-52690a8e728f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8"
Oct 30 22:36:31.090022 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:36:31.090006    4889 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-serving-cert: secret "maas-api-e2e-inf-a-d85da4-serving-cert" not found
Oct 30 22:36:31.090082 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:36:31.090072    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c660853-ad95-4bc2-b1f5-52690a8e728f-maas-api-tls podName:1c660853-ad95-4bc2-b1f5-52690a8e728f nodeName:}" failed. No retries permitted until 2026-10-30 22:36:31.590051639 +0000 UTC m=+1776.946417952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1c660853-ad95-4bc2-b1f5-52690a8e728f-maas-api-tls") pod "maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8" (UID: "1c660853-ad95-4bc2-b1f5-52690a8e728f") : secret "maas-api-e2e-inf-a-d85da4-serving-cert" not found
Oct 30 22:36:31.090407 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:31.090381    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c660853-ad95-4bc2-b1f5-52690a8e728f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8\" (UID: \"1c660853-ad95-4bc2-b1f5-52690a8e728f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8"
Oct 30 22:36:31.091313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:31.091283    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:31.092697 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:31.092677    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c660853-ad95-4bc2-b1f5-52690a8e728f-metrics-certs\") pod \"maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8\" (UID: \"1c660853-ad95-4bc2-b1f5-52690a8e728f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8"
Oct 30 22:36:31.165285 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:31.165188    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw9nx\" (UniqueName: \"kubernetes.io/projected/1c660853-ad95-4bc2-b1f5-52690a8e728f-kube-api-access-gw9nx\") pod \"maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8\" (UID: \"1c660853-ad95-4bc2-b1f5-52690a8e728f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8"
Oct 30 22:36:31.264252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:31.263817    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:31.436022 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:31.435923    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:31.628424 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:31.628384    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c660853-ad95-4bc2-b1f5-52690a8e728f-maas-api-tls\") pod \"maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8\" (UID: \"1c660853-ad95-4bc2-b1f5-52690a8e728f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8"
Oct 30 22:36:31.635472 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:31.635434    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c660853-ad95-4bc2-b1f5-52690a8e728f-maas-api-tls\") pod \"maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8\" (UID: \"1c660853-ad95-4bc2-b1f5-52690a8e728f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8"
Oct 30 22:36:31.885843 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:31.885811    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8"
Oct 30 22:36:32.467300 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:32.467274    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8"]
Oct 30 22:36:33.073377 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:33.073344    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8" event={"ID":"1c660853-ad95-4bc2-b1f5-52690a8e728f","Type":"ContainerStarted","Data":"4cf97bfabb8b3c3728f3644d53494e78645c1318eb3878a797f6435ecbbd7613"}
Oct 30 22:36:33.073734 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:33.073401    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8" event={"ID":"1c660853-ad95-4bc2-b1f5-52690a8e728f","Type":"ContainerStarted","Data":"548aab1d9df7dd91f13a8be98ebb88ae4c2fa658b8d73a2dc3165affe44db229"}
Oct 30 22:36:33.073734 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:33.073454    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8"
Oct 30 22:36:33.131854 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:33.131784    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8" podStartSLOduration=2.6495850389999998 podStartE2EDuration="3.131763701s" totalImagesPullingTime="482.178662ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:33.128674607 +0000 UTC m=+1778.485040924" watchObservedRunningTime="2026-10-30 22:36:33.131763701 +0000 UTC m=+1778.488130016"
Oct 30 22:36:33.389248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:33.388274    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:35.046906 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:35.045940    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:36.842782 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:36.842752    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:37.882554 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:37.881524    4889 patch_prober.go:29] interesting pod/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": read tcp 10.132.0.2:36230->10.132.0.163:15021: read: connection reset by peer" start-of-body=
Oct 30 22:36:37.882554 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:37.881589    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd" podUID="b0758d58-495a-4f8e-a1f6-daaca5024f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": read tcp 10.132.0.2:36230->10.132.0.163:15021: read: connection reset by peer"
Oct 30 22:36:38.133869 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:38.133755    4889 generic.go:356] "Generic (PLEG): container finished" podID="b0758d58-495a-4f8e-a1f6-daaca5024f8c" containerID="1441a09a6ac5bf01a605cb2ea241faee3b13fc3c8da4ae0cdb0c0ba560b991b6" exitCode=0
Oct 30 22:36:38.133869 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:38.133839    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd" event={"ID":"b0758d58-495a-4f8e-a1f6-daaca5024f8c","Type":"ContainerDied","Data":"1441a09a6ac5bf01a605cb2ea241faee3b13fc3c8da4ae0cdb0c0ba560b991b6"}
Oct 30 22:36:38.134380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:38.134363    4889 scope.go:122] "RemoveContainer" containerID="1441a09a6ac5bf01a605cb2ea241faee3b13fc3c8da4ae0cdb0c0ba560b991b6"
Oct 30 22:36:38.136203 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:38.136174    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:36:38.136321 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:38.136263    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:36:38.136321 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:38.136294    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:36:38.649087 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:38.649050    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:39.087696 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:39.087667    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8"
Oct 30 22:36:39.147034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:39.147002    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd" event={"ID":"b0758d58-495a-4f8e-a1f6-daaca5024f8c","Type":"ContainerStarted","Data":"ffc6f8ccb1ed86992599d55bbbe3f5a0caa42bcd4fda7cd67357c624a2bb9209"}
Oct 30 22:36:39.353008 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:39.352920    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:36:39.353199 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:39.353179    4889 patch_prober.go:29] interesting pod/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 22:36:39.353304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:39.353276    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd" podUID="b0758d58-495a-4f8e-a1f6-daaca5024f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 22:36:40.303263 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:36:40.303213    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:36:40.352917 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:40.352878    4889 patch_prober.go:29] interesting pod/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 22:36:40.353123 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:40.352950    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd" podUID="b0758d58-495a-4f8e-a1f6-daaca5024f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 22:36:40.643239 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:40.643132    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:41.352850 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:41.352813    4889 patch_prober.go:29] interesting pod/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 22:36:41.353278 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:41.352866    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd" podUID="b0758d58-495a-4f8e-a1f6-daaca5024f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 22:36:42.267608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:42.267566    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:42.353211 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:42.353177    4889 patch_prober.go:29] interesting pod/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 22:36:42.353633 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:42.353283    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd" podUID="b0758d58-495a-4f8e-a1f6-daaca5024f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 22:36:43.106584 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:43.106551    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:43.275451 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:43.275415    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:43.370309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:43.364849    4889 patch_prober.go:29] interesting pod/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 22:36:43.370309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:43.364920    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd" podUID="b0758d58-495a-4f8e-a1f6-daaca5024f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 22:36:43.448163 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:43.448132    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:43.791490 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:43.791455    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:44.353721 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:44.353681    4889 patch_prober.go:29] interesting pod/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 22:36:44.353923 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:44.353750    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd" podUID="b0758d58-495a-4f8e-a1f6-daaca5024f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 22:36:45.170776 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.170745    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"]
Oct 30 22:36:45.174913 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.174892    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.198618 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.198580    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"]
Oct 30 22:36:45.264870 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.264829    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istiod-ca-cert\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.265065 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.264949    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-data\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.265065 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.264992    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-podinfo\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.265186 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.265169    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-workload-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.265313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.265294    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-envoy\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.265356 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.265330    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-token\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.265418 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.265351    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-credential-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.265481 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.265459    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmlxk\" (UniqueName: \"kubernetes.io/projected/007bdaf6-3319-49db-850b-7fd9b87a5b7a-kube-api-access-vmlxk\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.265541 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.265503    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-workload-certs\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.369688 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.368683    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istiod-ca-cert\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.370038 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.370013    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-data\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.375000 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.374969    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:36:45.378311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.369647    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istiod-ca-cert\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.386337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.379482    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:36:45.386337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.370492    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-data\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.386337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.380216    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-podinfo\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.386337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.381098    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-workload-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.386337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.381172    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-envoy\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.386337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.381213    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-token\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.386337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.381271    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-credential-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.386337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.381329    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmlxk\" (UniqueName: \"kubernetes.io/projected/007bdaf6-3319-49db-850b-7fd9b87a5b7a-kube-api-access-vmlxk\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.386337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.381365    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-workload-certs\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.386337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.381924    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-workload-certs\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.386337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.382168    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-workload-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.386337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.385052    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:36:45.386337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.385711    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:45.386337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.386086    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-credential-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.419360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.414885    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-envoy\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.445029 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.444377    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-podinfo\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.478318 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.477860    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmlxk\" (UniqueName: \"kubernetes.io/projected/007bdaf6-3319-49db-850b-7fd9b87a5b7a-kube-api-access-vmlxk\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.478494 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.478323    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-token\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.489268 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.489217    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:45.838467 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.838428    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"]
Oct 30 22:36:45.838886 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:36:45.838862    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007bdaf6_3319_49db_850b_7fd9b87a5b7a.slice/crio-bb6838e1d46e5ead12096ae1c6398979f74ec2b4b962c8f7c52b71947ad69e7b WatchSource:0}: Error finding container bb6838e1d46e5ead12096ae1c6398979f74ec2b4b962c8f7c52b71947ad69e7b: Status 404 returned error can't find the container with id bb6838e1d46e5ead12096ae1c6398979f74ec2b4b962c8f7c52b71947ad69e7b
Oct 30 22:36:45.842148 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.842113    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:36:45.842347 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.842184    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:36:45.842347 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:45.842212    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:36:46.222097 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:46.221994    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795" event={"ID":"007bdaf6-3319-49db-850b-7fd9b87a5b7a","Type":"ContainerStarted","Data":"4d6e6d920484364b8ef55f8cf0213339ad6e8348219075bc3d2300855cb95f85"}
Oct 30 22:36:46.222620 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:46.222107    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795" event={"ID":"007bdaf6-3319-49db-850b-7fd9b87a5b7a","Type":"ContainerStarted","Data":"bb6838e1d46e5ead12096ae1c6398979f74ec2b4b962c8f7c52b71947ad69e7b"}
Oct 30 22:36:46.274422 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:46.274365    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795" podStartSLOduration=1.274335727 podStartE2EDuration="1.274335727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:46.27150954 +0000 UTC m=+1791.627875866" watchObservedRunningTime="2026-10-30 22:36:46.274335727 +0000 UTC m=+1791.630702042"
Oct 30 22:36:46.490352 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:46.490251    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:46.491865 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:46.491836    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 30 22:36:46.491988 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:46.491891    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795" podUID="007bdaf6-3319-49db-850b-7fd9b87a5b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 30 22:36:47.340250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:47.317693    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:47.490382 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:47.490349    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 30 22:36:47.490566 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:47.490405    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795" podUID="007bdaf6-3319-49db-850b-7fd9b87a5b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 30 22:36:48.416260 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:48.406367    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:48.509312 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:48.506694    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 30 22:36:48.509312 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:48.506758    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795" podUID="007bdaf6-3319-49db-850b-7fd9b87a5b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 30 22:36:49.293279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:49.291204    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:49.490236 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:49.490191    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 30 22:36:49.490667 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:49.490264    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795" podUID="007bdaf6-3319-49db-850b-7fd9b87a5b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 30 22:36:50.490419 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:50.490382    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 30 22:36:50.490842 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:50.490445    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795" podUID="007bdaf6-3319-49db-850b-7fd9b87a5b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 30 22:36:50.570946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:50.570900    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:50.762189 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:50.762152    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:50.797129 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:50.797092    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"]
Oct 30 22:36:50.797458 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:50.797385    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj" podUID="41d10d03-c9af-4c80-b5c5-692d1a3953af" containerName="istio-proxy" containerID="cri-o://89f6e908963789f1463396659e44881b9816d9f25db9e9d1fb22e47b7a3b1884" gracePeriod=30
Oct 30 22:36:51.089304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:51.087163    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:51.493123 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:51.493089    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 30 22:36:51.494432 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:51.493869    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795" podUID="007bdaf6-3319-49db-850b-7fd9b87a5b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 30 22:36:51.496050 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:51.496028    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:51.731799 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:51.731764    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:52.502434 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:52.502397    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:52.506444 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:52.504842    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 30 22:36:52.506444 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:52.504895    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795" podUID="007bdaf6-3319-49db-850b-7fd9b87a5b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 30 22:36:52.754002 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:52.753912    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:53.494735 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:53.494704    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:53.495242 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:53.495206    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:53.495996 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:53.495972    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:36:53.800921 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:53.800807    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:54.158737 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:54.158644    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:55.483888 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:55.483855    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 22:36:55.484323 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:55.484282    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 22:36:55.772772 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:55.772736    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:56.164685 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.164653    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:36:56.258247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.256096    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-data\") pod \"41d10d03-c9af-4c80-b5c5-692d1a3953af\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") "
Oct 30 22:36:56.258247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.256173    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-podinfo\") pod \"41d10d03-c9af-4c80-b5c5-692d1a3953af\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") "
Oct 30 22:36:56.258247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.256325    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-workload-socket\") pod \"41d10d03-c9af-4c80-b5c5-692d1a3953af\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") "
Oct 30 22:36:56.258247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.256419    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-credential-socket\") pod \"41d10d03-c9af-4c80-b5c5-692d1a3953af\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") "
Oct 30 22:36:56.258247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.256480    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-envoy\") pod \"41d10d03-c9af-4c80-b5c5-692d1a3953af\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") "
Oct 30 22:36:56.258247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.256615    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-workload-certs\") pod \"41d10d03-c9af-4c80-b5c5-692d1a3953af\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") "
Oct 30 22:36:56.258247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.256683    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41d10d03-c9af-4c80-b5c5-692d1a3953af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d10d03-c9af-4c80-b5c5-692d1a3953af-istiod-ca-cert\") pod \"41d10d03-c9af-4c80-b5c5-692d1a3953af\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") "
Oct 30 22:36:56.258247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.256797    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41d10d03-c9af-4c80-b5c5-692d1a3953af-kube-api-access-nzgpx\" (UniqueName: \"kubernetes.io/projected/41d10d03-c9af-4c80-b5c5-692d1a3953af-kube-api-access-nzgpx\") pod \"41d10d03-c9af-4c80-b5c5-692d1a3953af\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") "
Oct 30 22:36:56.258247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.256841    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-token\" (UniqueName: \"kubernetes.io/projected/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-token\") pod \"41d10d03-c9af-4c80-b5c5-692d1a3953af\" (UID: \"41d10d03-c9af-4c80-b5c5-692d1a3953af\") "
Oct 30 22:36:56.258832 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.258334    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-credential-socket" pod "41d10d03-c9af-4c80-b5c5-692d1a3953af" (UID: "41d10d03-c9af-4c80-b5c5-692d1a3953af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:56.258832 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.258473    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-workload-certs" pod "41d10d03-c9af-4c80-b5c5-692d1a3953af" (UID: "41d10d03-c9af-4c80-b5c5-692d1a3953af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:56.258832 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.258577    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-workload-socket" pod "41d10d03-c9af-4c80-b5c5-692d1a3953af" (UID: "41d10d03-c9af-4c80-b5c5-692d1a3953af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:56.259075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.259043    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d10d03-c9af-4c80-b5c5-692d1a3953af-istiod-ca-cert" pod "41d10d03-c9af-4c80-b5c5-692d1a3953af" (UID: "41d10d03-c9af-4c80-b5c5-692d1a3953af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:56.260430 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.259432    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-data" pod "41d10d03-c9af-4c80-b5c5-692d1a3953af" (UID: "41d10d03-c9af-4c80-b5c5-692d1a3953af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:56.269279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.269250    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-token" pod "41d10d03-c9af-4c80-b5c5-692d1a3953af" (UID: "41d10d03-c9af-4c80-b5c5-692d1a3953af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:56.269554 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.269528    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d10d03-c9af-4c80-b5c5-692d1a3953af-kube-api-access-nzgpx" pod "41d10d03-c9af-4c80-b5c5-692d1a3953af" (UID: "41d10d03-c9af-4c80-b5c5-692d1a3953af"). InnerVolumeSpecName "kube-api-access-nzgpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:56.269721 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.269695    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-envoy" pod "41d10d03-c9af-4c80-b5c5-692d1a3953af" (UID: "41d10d03-c9af-4c80-b5c5-692d1a3953af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:56.272711 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.272680    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-podinfo" pod "41d10d03-c9af-4c80-b5c5-692d1a3953af" (UID: "41d10d03-c9af-4c80-b5c5-692d1a3953af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:56.360121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.360088    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:56.360121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.360119    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:56.360121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.360129    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:56.360121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.360137    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:56.360414 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.360148    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:56.360414 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.360157    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d10d03-c9af-4c80-b5c5-692d1a3953af-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:56.360414 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.360165    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzgpx\" (UniqueName: \"kubernetes.io/projected/41d10d03-c9af-4c80-b5c5-692d1a3953af-kube-api-access-nzgpx\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:56.360414 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.360174    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:56.360414 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.360182    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d10d03-c9af-4c80-b5c5-692d1a3953af-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:36:56.387752 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.387678    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:56.607256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.604405    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:56.726295 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.711545    4889 generic.go:356] "Generic (PLEG): container finished" podID="41d10d03-c9af-4c80-b5c5-692d1a3953af" containerID="89f6e908963789f1463396659e44881b9816d9f25db9e9d1fb22e47b7a3b1884" exitCode=0
Oct 30 22:36:56.726295 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.711758    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj" event={"ID":"41d10d03-c9af-4c80-b5c5-692d1a3953af","Type":"ContainerDied","Data":"89f6e908963789f1463396659e44881b9816d9f25db9e9d1fb22e47b7a3b1884"}
Oct 30 22:36:56.726295 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.711844    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj" event={"ID":"41d10d03-c9af-4c80-b5c5-692d1a3953af","Type":"ContainerDied","Data":"ccd21f15d77f00099b71bce368f7a4ef9434dcda95fc4b9ee6414a5ee2a66e3b"}
Oct 30 22:36:56.726295 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.711878    4889 scope.go:122] "RemoveContainer" containerID="89f6e908963789f1463396659e44881b9816d9f25db9e9d1fb22e47b7a3b1884"
Oct 30 22:36:56.726295 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.712060    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"
Oct 30 22:36:56.769565 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.769530    4889 scope.go:122] "RemoveContainer" containerID="4135272e3e3db1908e8789c1be14aff438670105bfaebf377c19742d1df753d3"
Oct 30 22:36:56.782744 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.782712    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:56.799689 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.799664    4889 scope.go:122] "RemoveContainer" containerID="89f6e908963789f1463396659e44881b9816d9f25db9e9d1fb22e47b7a3b1884"
Oct 30 22:36:56.800070 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:36:56.800046    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89f6e908963789f1463396659e44881b9816d9f25db9e9d1fb22e47b7a3b1884\": container with ID starting with 89f6e908963789f1463396659e44881b9816d9f25db9e9d1fb22e47b7a3b1884 not found: ID does not exist" containerID="89f6e908963789f1463396659e44881b9816d9f25db9e9d1fb22e47b7a3b1884"
Oct 30 22:36:56.800168 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.800083    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89f6e908963789f1463396659e44881b9816d9f25db9e9d1fb22e47b7a3b1884"} err="failed to get container status \"89f6e908963789f1463396659e44881b9816d9f25db9e9d1fb22e47b7a3b1884\": rpc error: code = NotFound desc = could not find container \"89f6e908963789f1463396659e44881b9816d9f25db9e9d1fb22e47b7a3b1884\": container with ID starting with 89f6e908963789f1463396659e44881b9816d9f25db9e9d1fb22e47b7a3b1884 not found: ID does not exist"
Oct 30 22:36:56.800168 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.800108    4889 scope.go:122] "RemoveContainer" containerID="4135272e3e3db1908e8789c1be14aff438670105bfaebf377c19742d1df753d3"
Oct 30 22:36:56.800391 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:36:56.800370    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4135272e3e3db1908e8789c1be14aff438670105bfaebf377c19742d1df753d3\": container with ID starting with 4135272e3e3db1908e8789c1be14aff438670105bfaebf377c19742d1df753d3 not found: ID does not exist" containerID="4135272e3e3db1908e8789c1be14aff438670105bfaebf377c19742d1df753d3"
Oct 30 22:36:56.800445 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.800403    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4135272e3e3db1908e8789c1be14aff438670105bfaebf377c19742d1df753d3"} err="failed to get container status \"4135272e3e3db1908e8789c1be14aff438670105bfaebf377c19742d1df753d3\": rpc error: code = NotFound desc = could not find container \"4135272e3e3db1908e8789c1be14aff438670105bfaebf377c19742d1df753d3\": container with ID starting with 4135272e3e3db1908e8789c1be14aff438670105bfaebf377c19742d1df753d3 not found: ID does not exist"
Oct 30 22:36:56.817872 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.817838    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"]
Oct 30 22:36:56.821943 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:56.821914    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4f7439-openshift-default-84f666696f-7wlcj"]
Oct 30 22:36:57.041008 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.040968    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:57.408367 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.408275    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"]
Oct 30 22:36:57.409551 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.409521    4889 cpu_manager.go:414] "containerMap: removing container" podUID="41d10d03-c9af-4c80-b5c5-692d1a3953af" containerName="istio-proxy"
Oct 30 22:36:57.409551 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.409550    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41d10d03-c9af-4c80-b5c5-692d1a3953af" containerName="istio-proxy"
Oct 30 22:36:57.410324 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.409809    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="41d10d03-c9af-4c80-b5c5-692d1a3953af" containerName="istio-proxy"
Oct 30 22:36:57.416449 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.416427    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.450878 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.450844    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"]
Oct 30 22:36:57.451177 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.451066    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-54c7f0-openshift-default-dockercfg-vj8jm\""
Oct 30 22:36:57.501276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.499700    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl8lx\" (UniqueName: \"kubernetes.io/projected/2ced92c8-b2b3-4d29-86ca-b816484d7b69-kube-api-access-sl8lx\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.501276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.499790    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-credential-socket\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.501276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.499832    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-workload-certs\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.501276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.499857    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-data\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.501276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.499891    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-workload-socket\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.501276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.500080    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-podinfo\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.501276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.500111    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-envoy\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.501276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.500300    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istiod-ca-cert\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.501276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.500347    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-token\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.540648 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.540611    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:57.601866 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.601817    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istiod-ca-cert\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.601866 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.601863    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-token\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.602133 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.601975    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl8lx\" (UniqueName: \"kubernetes.io/projected/2ced92c8-b2b3-4d29-86ca-b816484d7b69-kube-api-access-sl8lx\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.602133 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.602012    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-credential-socket\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.602133 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.602053    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-workload-certs\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.602133 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.602078    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-data\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.602133 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.602127    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-workload-socket\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.602422 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.602343    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-podinfo\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.602422 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.602383    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-envoy\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.602541 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.602470    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-credential-socket\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.602541 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.602502    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-data\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.602701 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.602597    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-workload-socket\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.602746 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.602713    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-workload-certs\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.603020 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.603001    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istiod-ca-cert\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.604876 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.604855    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-envoy\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.604992 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.604974    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-podinfo\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.625746 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.625723    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl8lx\" (UniqueName: \"kubernetes.io/projected/2ced92c8-b2b3-4d29-86ca-b816484d7b69-kube-api-access-sl8lx\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.626082 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.625972    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-token\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:57.766284 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:57.760913    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:58.268138 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:58.268105    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"]
Oct 30 22:36:58.269722 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:36:58.269655    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ced92c8_b2b3_4d29_86ca_b816484d7b69.slice/crio-37aa12b0e0f16e96a5530cbedd476094b19b7a8e8428d88d12c3256833b348c0 WatchSource:0}: Error finding container 37aa12b0e0f16e96a5530cbedd476094b19b7a8e8428d88d12c3256833b348c0: Status 404 returned error can't find the container with id 37aa12b0e0f16e96a5530cbedd476094b19b7a8e8428d88d12c3256833b348c0
Oct 30 22:36:58.275121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:58.273843    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:36:58.275121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:58.273928    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:36:58.275121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:58.273970    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:36:58.801245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:58.790817    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41d10d03-c9af-4c80-b5c5-692d1a3953af" path="/var/lib/kubelet/pods/41d10d03-c9af-4c80-b5c5-692d1a3953af/volumes"
Oct 30 22:36:58.801245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:58.797286    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d"]
Oct 30 22:36:58.801245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:58.798586    4889 cpu_manager.go:414] "containerMap: removing container" podUID="41d10d03-c9af-4c80-b5c5-692d1a3953af" containerName="istio-proxy"
Oct 30 22:36:58.801245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:58.798606    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41d10d03-c9af-4c80-b5c5-692d1a3953af" containerName="istio-proxy"
Oct 30 22:36:58.801245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:58.798796    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="41d10d03-c9af-4c80-b5c5-692d1a3953af" containerName="istio-proxy"
Oct 30 22:36:58.804747 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:58.804723    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d"]
Oct 30 22:36:58.805054 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:58.804998    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d"
Oct 30 22:36:58.820821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:58.820790    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:58.827627 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:58.826912    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-bc48cc\""
Oct 30 22:36:58.831728 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:58.831644    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz" event={"ID":"2ced92c8-b2b3-4d29-86ca-b816484d7b69","Type":"ContainerStarted","Data":"1d5c8fa33d8ec6ac2d3999d9ce57aed5d988289a718cc4c15c5778a4cc654c5e"}
Oct 30 22:36:58.831728 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:58.831700    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz" event={"ID":"2ced92c8-b2b3-4d29-86ca-b816484d7b69","Type":"ContainerStarted","Data":"37aa12b0e0f16e96a5530cbedd476094b19b7a8e8428d88d12c3256833b348c0"}
Oct 30 22:36:58.848855 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:58.848822    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-bc48cc-dockercfg-rgj97\""
Oct 30 22:36:58.908417 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:58.908346    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz" podStartSLOduration=1.908328734 podStartE2EDuration="1.908328734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:58.903175841 +0000 UTC m=+1804.259542169" watchObservedRunningTime="2026-10-30 22:36:58.908328734 +0000 UTC m=+1804.264695051"
Oct 30 22:36:58.920500 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:58.920464    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxtcm\" (UniqueName: \"kubernetes.io/projected/9d8d205a-47b3-45f7-9262-3083ea98e5c9-kube-api-access-dxtcm\") pod \"payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d\" (UID: \"9d8d205a-47b3-45f7-9262-3083ea98e5c9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d"
Oct 30 22:36:58.920668 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:58.920645    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d8d205a-47b3-45f7-9262-3083ea98e5c9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d\" (UID: \"9d8d205a-47b3-45f7-9262-3083ea98e5c9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d"
Oct 30 22:36:59.022033 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:59.021993    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d8d205a-47b3-45f7-9262-3083ea98e5c9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d\" (UID: \"9d8d205a-47b3-45f7-9262-3083ea98e5c9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d"
Oct 30 22:36:59.022330 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:59.022312    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxtcm\" (UniqueName: \"kubernetes.io/projected/9d8d205a-47b3-45f7-9262-3083ea98e5c9-kube-api-access-dxtcm\") pod \"payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d\" (UID: \"9d8d205a-47b3-45f7-9262-3083ea98e5c9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d"
Oct 30 22:36:59.022771 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:59.022747    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d8d205a-47b3-45f7-9262-3083ea98e5c9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d\" (UID: \"9d8d205a-47b3-45f7-9262-3083ea98e5c9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d"
Oct 30 22:36:59.044647 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:59.044611    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxtcm\" (UniqueName: \"kubernetes.io/projected/9d8d205a-47b3-45f7-9262-3083ea98e5c9-kube-api-access-dxtcm\") pod \"payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d\" (UID: \"9d8d205a-47b3-45f7-9262-3083ea98e5c9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d"
Oct 30 22:36:59.151370 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:59.148539    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d"
Oct 30 22:36:59.496656 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:36:59.495535    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8d205a_47b3_45f7_9262_3083ea98e5c9.slice/crio-6e4b668f517f40084e80023b8b33b9b4279b54afbf6100dd710ca42a00242836 WatchSource:0}: Error finding container 6e4b668f517f40084e80023b8b33b9b4279b54afbf6100dd710ca42a00242836: Status 404 returned error can't find the container with id 6e4b668f517f40084e80023b8b33b9b4279b54afbf6100dd710ca42a00242836
Oct 30 22:36:59.496656 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:59.496548    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d"]
Oct 30 22:36:59.533009 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:59.532977    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:59.772623 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:59.770946    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:59.817794 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:59.817761    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:59.860100 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:59.860062    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d" event={"ID":"9d8d205a-47b3-45f7-9262-3083ea98e5c9","Type":"ContainerStarted","Data":"225e39ef4eeeca381311b5a437273cca675472a01372f933060111d7367cda90"}
Oct 30 22:36:59.860356 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:59.860117    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d"
Oct 30 22:36:59.860710 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:59.860412    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d" event={"ID":"9d8d205a-47b3-45f7-9262-3083ea98e5c9","Type":"ContainerStarted","Data":"6e4b668f517f40084e80023b8b33b9b4279b54afbf6100dd710ca42a00242836"}
Oct 30 22:36:59.860796 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:59.860736    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:59.870293 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:59.868455    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:36:59.914516 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:36:59.914461    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d" podStartSLOduration=1.914446077 podStartE2EDuration="1.914446077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:59.910792972 +0000 UTC m=+1805.267159289" watchObservedRunningTime="2026-10-30 22:36:59.914446077 +0000 UTC m=+1805.270812421"
Oct 30 22:37:00.261661 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:00.253896    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:00.567412 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:00.567331    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:00.740271 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:00.737086    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:00.992121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:00.992085    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:01.173517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:01.173483    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:01.428529 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:01.428443    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:02.142534 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:02.142496    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:02.345376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:02.339383    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:02.936111 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:02.936075    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:04.409238 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:04.409195    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:04.595484 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:04.595428    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:04.913250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:04.844833    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:06.825907 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:06.820712    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:07.595138 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:07.595097    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:07.764127 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:07.764095    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:07.967306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:07.964778    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:08.724246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:08.723938    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:08.898352 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:08.898319    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:09.235729 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:09.235693    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:09.413576 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:09.413542    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:10.304823 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:37:10.304791    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:37:10.644368 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:10.644046    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:11.890200 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:11.890173    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d"
Oct 30 22:37:12.666962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:12.666928    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:12.840101 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:12.840065    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:13.026373 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:13.026333    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:14.555846 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:14.555796    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:14.724700 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:14.724662    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:16.695673 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:16.695637    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:18.649333 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:18.639628    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:20.558250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:20.553673    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:22.254531 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:22.254491    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:22.647280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:22.641747    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:22.817667 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:22.817621    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:23.239381 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:23.239344    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:23.845923 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:23.845885    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:24.717944 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:24.717910    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:26.758430 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:26.758386    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:27.261920 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:27.261874    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:27.441725 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:27.441692    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:29.040852 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:29.040803    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:29.265886 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:29.265829    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:29.416572 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:29.416470    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4"]
Oct 30 22:37:29.421877 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:29.421845    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4"
Oct 30 22:37:29.460084 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:29.459995    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:29.460742 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:29.460713    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4"]
Oct 30 22:37:29.472614 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:29.472580    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx"]
Oct 30 22:37:29.486249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:29.482966    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f0bb45\""
Oct 30 22:37:29.486249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:29.483185    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f0bb45-dockercfg-rb5j2\""
Oct 30 22:37:29.497113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:29.497087    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx"
Oct 30 22:37:29.587104 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:29.514489    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4\" (UID: \"9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4"
Oct 30 22:37:29.587104 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:29.583610    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzxcl\" (UniqueName: \"kubernetes.io/projected/9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81-kube-api-access-hzxcl\") pod \"payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4\" (UID: \"9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4"
Oct 30 22:37:29.605249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:29.601141    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx"]
Oct 30 22:37:29.744337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:29.692518    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4\" (UID: \"9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4"
Oct 30 22:37:29.744337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:29.692664    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cztlc\" (UniqueName: \"kubernetes.io/projected/8789eb93-f28c-4fca-9b60-45811622383f-kube-api-access-cztlc\") pod \"payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx\" (UID: \"8789eb93-f28c-4fca-9b60-45811622383f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx"
Oct 30 22:37:29.744337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:29.692850    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8789eb93-f28c-4fca-9b60-45811622383f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx\" (UID: \"8789eb93-f28c-4fca-9b60-45811622383f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx"
Oct 30 22:37:29.744337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:29.692907    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxcl\" (UniqueName: \"kubernetes.io/projected/9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81-kube-api-access-hzxcl\") pod \"payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4\" (UID: \"9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4"
Oct 30 22:37:29.744337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:29.694445    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4\" (UID: \"9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4"
Oct 30 22:37:29.813635 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:29.802543    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8789eb93-f28c-4fca-9b60-45811622383f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx\" (UID: \"8789eb93-f28c-4fca-9b60-45811622383f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx"
Oct 30 22:37:29.813635 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:29.802858    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cztlc\" (UniqueName: \"kubernetes.io/projected/8789eb93-f28c-4fca-9b60-45811622383f-kube-api-access-cztlc\") pod \"payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx\" (UID: \"8789eb93-f28c-4fca-9b60-45811622383f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx"
Oct 30 22:37:29.813635 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:29.804349    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8789eb93-f28c-4fca-9b60-45811622383f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx\" (UID: \"8789eb93-f28c-4fca-9b60-45811622383f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx"
Oct 30 22:37:29.833190 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:29.828923    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzxcl\" (UniqueName: \"kubernetes.io/projected/9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81-kube-api-access-hzxcl\") pod \"payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4\" (UID: \"9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4"
Oct 30 22:37:29.852092 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:29.852034    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cztlc\" (UniqueName: \"kubernetes.io/projected/8789eb93-f28c-4fca-9b60-45811622383f-kube-api-access-cztlc\") pod \"payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx\" (UID: \"8789eb93-f28c-4fca-9b60-45811622383f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx"
Oct 30 22:37:30.118187 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:30.118097    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4"
Oct 30 22:37:30.151592 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:30.151563    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx"
Oct 30 22:37:30.439139 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:37:30.439089    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af1fe41_3e2a_4beb_8f28_0e6a3ecbdb81.slice/crio-15bfd3af80e24e4bada8f1b816baf9b4f1d2f8bcdfb4297316d3aa034cfed5b0 WatchSource:0}: Error finding container 15bfd3af80e24e4bada8f1b816baf9b4f1d2f8bcdfb4297316d3aa034cfed5b0: Status 404 returned error can't find the container with id 15bfd3af80e24e4bada8f1b816baf9b4f1d2f8bcdfb4297316d3aa034cfed5b0
Oct 30 22:37:30.440978 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:30.440939    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4"]
Oct 30 22:37:30.494076 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:30.494042    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx"]
Oct 30 22:37:30.494528 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:37:30.494497    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8789eb93_f28c_4fca_9b60_45811622383f.slice/crio-320d3156c8e39f6fc8123f5eb0d263d902ca7a51b4766b338ec608ddd19b6fee WatchSource:0}: Error finding container 320d3156c8e39f6fc8123f5eb0d263d902ca7a51b4766b338ec608ddd19b6fee: Status 404 returned error can't find the container with id 320d3156c8e39f6fc8123f5eb0d263d902ca7a51b4766b338ec608ddd19b6fee
Oct 30 22:37:30.621386 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:30.621341    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4" event={"ID":"9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81","Type":"ContainerStarted","Data":"585683b43d52947b987bd550f25e9986912524f74978f52873fb7f83489d7908"}
Oct 30 22:37:30.621574 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:30.621409    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4" event={"ID":"9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81","Type":"ContainerStarted","Data":"15bfd3af80e24e4bada8f1b816baf9b4f1d2f8bcdfb4297316d3aa034cfed5b0"}
Oct 30 22:37:30.621574 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:30.621456    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4"
Oct 30 22:37:30.623502 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:30.623468    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx" event={"ID":"8789eb93-f28c-4fca-9b60-45811622383f","Type":"ContainerStarted","Data":"299a7990198fdb43910c953cdf671ddfcfcf347caca113157646f3c703beaeb3"}
Oct 30 22:37:30.623656 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:30.623526    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx" event={"ID":"8789eb93-f28c-4fca-9b60-45811622383f","Type":"ContainerStarted","Data":"320d3156c8e39f6fc8123f5eb0d263d902ca7a51b4766b338ec608ddd19b6fee"}
Oct 30 22:37:30.623656 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:30.623565    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx"
Oct 30 22:37:30.675606 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:30.675549    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4" podStartSLOduration=1.675534952 podStartE2EDuration="1.675534952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:30.673115488 +0000 UTC m=+1836.029481816" watchObservedRunningTime="2026-10-30 22:37:30.675534952 +0000 UTC m=+1836.031901266"
Oct 30 22:37:30.722801 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:30.722697    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx" podStartSLOduration=1.72268159 podStartE2EDuration="1.72268159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:30.720812296 +0000 UTC m=+1836.077178623" watchObservedRunningTime="2026-10-30 22:37:30.72268159 +0000 UTC m=+1836.079047905"
Oct 30 22:37:31.465571 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:31.460959    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:32.815341 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:32.815302    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"]
Oct 30 22:37:32.822856 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:32.822826    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"
Oct 30 22:37:32.840950 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:32.840918    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"]
Oct 30 22:37:32.846241 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:32.846197    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6013c8\"/\"kube-root-ca.crt\""
Oct 30 22:37:32.846374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:32.846285    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6013c8\"/\"openshift-service-ca.crt\""
Oct 30 22:37:32.855359 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:32.855336    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6013c8\"/\"default-dockercfg-w6fv9\""
Oct 30 22:37:32.855474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:32.855382    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6013c8\"/\"test-model-6013c8-kserve-self-signed-certs\""
Oct 30 22:37:32.930204 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:32.930171    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:32.981415 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:32.981355    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d863aec2-a41e-44bb-a09d-856643916b65-tls-certs\") pod \"test-model-6013c8-kserve-85fcbf9bc7-chrjl\" (UID: \"d863aec2-a41e-44bb-a09d-856643916b65\") " pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"
Oct 30 22:37:32.981637 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:32.981609    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-model-cache\") pod \"test-model-6013c8-kserve-85fcbf9bc7-chrjl\" (UID: \"d863aec2-a41e-44bb-a09d-856643916b65\") " pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"
Oct 30 22:37:32.981766 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:32.981743    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-dshm\") pod \"test-model-6013c8-kserve-85fcbf9bc7-chrjl\" (UID: \"d863aec2-a41e-44bb-a09d-856643916b65\") " pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"
Oct 30 22:37:32.981825 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:32.981800    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lppqf\" (UniqueName: \"kubernetes.io/projected/d863aec2-a41e-44bb-a09d-856643916b65-kube-api-access-lppqf\") pod \"test-model-6013c8-kserve-85fcbf9bc7-chrjl\" (UID: \"d863aec2-a41e-44bb-a09d-856643916b65\") " pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"
Oct 30 22:37:32.981883 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:32.981830    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-tmp-dir\") pod \"test-model-6013c8-kserve-85fcbf9bc7-chrjl\" (UID: \"d863aec2-a41e-44bb-a09d-856643916b65\") " pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"
Oct 30 22:37:32.982003 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:32.981988    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-home\") pod \"test-model-6013c8-kserve-85fcbf9bc7-chrjl\" (UID: \"d863aec2-a41e-44bb-a09d-856643916b65\") " pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"
Oct 30 22:37:33.086567 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:33.086484    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-model-cache\") pod \"test-model-6013c8-kserve-85fcbf9bc7-chrjl\" (UID: \"d863aec2-a41e-44bb-a09d-856643916b65\") " pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"
Oct 30 22:37:33.088152 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:33.088122    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-dshm\") pod \"test-model-6013c8-kserve-85fcbf9bc7-chrjl\" (UID: \"d863aec2-a41e-44bb-a09d-856643916b65\") " pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"
Oct 30 22:37:33.088907 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:33.088886    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lppqf\" (UniqueName: \"kubernetes.io/projected/d863aec2-a41e-44bb-a09d-856643916b65-kube-api-access-lppqf\") pod \"test-model-6013c8-kserve-85fcbf9bc7-chrjl\" (UID: \"d863aec2-a41e-44bb-a09d-856643916b65\") " pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"
Oct 30 22:37:33.089386 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:33.089367    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-tmp-dir\") pod \"test-model-6013c8-kserve-85fcbf9bc7-chrjl\" (UID: \"d863aec2-a41e-44bb-a09d-856643916b65\") " pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"
Oct 30 22:37:33.089906 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:33.089889    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-home\") pod \"test-model-6013c8-kserve-85fcbf9bc7-chrjl\" (UID: \"d863aec2-a41e-44bb-a09d-856643916b65\") " pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"
Oct 30 22:37:33.094559 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:33.094539    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d863aec2-a41e-44bb-a09d-856643916b65-tls-certs\") pod \"test-model-6013c8-kserve-85fcbf9bc7-chrjl\" (UID: \"d863aec2-a41e-44bb-a09d-856643916b65\") " pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"
Oct 30 22:37:33.110037 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:33.087138    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-model-cache\") pod \"test-model-6013c8-kserve-85fcbf9bc7-chrjl\" (UID: \"d863aec2-a41e-44bb-a09d-856643916b65\") " pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"
Oct 30 22:37:33.111020 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:33.094311    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-home\") pod \"test-model-6013c8-kserve-85fcbf9bc7-chrjl\" (UID: \"d863aec2-a41e-44bb-a09d-856643916b65\") " pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"
Oct 30 22:37:33.111256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:33.089779    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-tmp-dir\") pod \"test-model-6013c8-kserve-85fcbf9bc7-chrjl\" (UID: \"d863aec2-a41e-44bb-a09d-856643916b65\") " pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"
Oct 30 22:37:33.111395 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:33.098280    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-dshm\") pod \"test-model-6013c8-kserve-85fcbf9bc7-chrjl\" (UID: \"d863aec2-a41e-44bb-a09d-856643916b65\") " pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"
Oct 30 22:37:33.111495 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:33.103063    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d863aec2-a41e-44bb-a09d-856643916b65-tls-certs\") pod \"test-model-6013c8-kserve-85fcbf9bc7-chrjl\" (UID: \"d863aec2-a41e-44bb-a09d-856643916b65\") " pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"
Oct 30 22:37:33.128265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:33.126377    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lppqf\" (UniqueName: \"kubernetes.io/projected/d863aec2-a41e-44bb-a09d-856643916b65-kube-api-access-lppqf\") pod \"test-model-6013c8-kserve-85fcbf9bc7-chrjl\" (UID: \"d863aec2-a41e-44bb-a09d-856643916b65\") " pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"
Oct 30 22:37:33.151254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:33.143549    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"
Oct 30 22:37:33.444187 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:33.443116    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:33.552388 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:37:33.552357    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd863aec2_a41e_44bb_a09d_856643916b65.slice/crio-f4a1ddb45d9f0c32427e8d21a19e4af4f023f1c4af07cac3426714fd249d7e4e WatchSource:0}: Error finding container f4a1ddb45d9f0c32427e8d21a19e4af4f023f1c4af07cac3426714fd249d7e4e: Status 404 returned error can't find the container with id f4a1ddb45d9f0c32427e8d21a19e4af4f023f1c4af07cac3426714fd249d7e4e
Oct 30 22:37:33.552838 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:33.552812    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"]
Oct 30 22:37:33.676086 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:33.676050    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl" event={"ID":"d863aec2-a41e-44bb-a09d-856643916b65","Type":"ContainerStarted","Data":"abc9f778764a4fed983e27ac911d9d7edefa3ab49702d7c5a15c666112cd8ed6"}
Oct 30 22:37:33.676310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:33.676097    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl" event={"ID":"d863aec2-a41e-44bb-a09d-856643916b65","Type":"ContainerStarted","Data":"f4a1ddb45d9f0c32427e8d21a19e4af4f023f1c4af07cac3426714fd249d7e4e"}
Oct 30 22:37:33.725086 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:33.724982    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl" podStartSLOduration=1.724967748 podStartE2EDuration="1.724967748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:33.723539974 +0000 UTC m=+1839.079906288" watchObservedRunningTime="2026-10-30 22:37:33.724967748 +0000 UTC m=+1839.081334062"
Oct 30 22:37:34.144893 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:34.144863    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"
Oct 30 22:37:35.505750 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:35.505657    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:37.198747 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:37.195122    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:38.877151 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:38.873702    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:40.303271 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:37:40.303199    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:37:40.680947 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:40.680565    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:41.634104 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:41.634075    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4"
Oct 30 22:37:41.634539 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:41.634147    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx"
Oct 30 22:37:42.556455 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:42.556411    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:42.757060 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:42.757028    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:43.144475 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:43.144435    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"
Oct 30 22:37:43.158156 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:43.158129    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"
Oct 30 22:37:43.808684 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:43.808654    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"
Oct 30 22:37:44.505636 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:44.505590    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:46.254374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:46.249335    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:48.060880 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:48.060838    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:48.498643 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:48.498606    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:49.311498 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:49.311465    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:49.502887 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:49.502851    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:49.933294 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:49.933216    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:50.096511 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:50.096477    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:50.292111 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:50.292066    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:52.031365 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:52.031280    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:53.759930 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:53.759890    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:53.942555 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:53.942517    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:54.396325 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:54.396292    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:55.197248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:55.195041    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:55.532117 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:55.532085    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:55.879293 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:55.879200    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:57.657247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:57.649531    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:58.131697 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:58.131661    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:58.302575 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:58.302543    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:58.833950 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:58.833917    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:59.005928 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:59.005891    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:59.669537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:37:59.669501    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:00.545384 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:00.545349    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:00.593197 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:00.593162    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5"]
Oct 30 22:38:00.599483 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:00.599465    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5"
Oct 30 22:38:00.620746 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:00.620715    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5"]
Oct 30 22:38:00.625289 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:00.623024    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c13db0\""
Oct 30 22:38:00.697539 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:00.697504    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fghmj\" (UniqueName: \"kubernetes.io/projected/bf02b0cb-f674-4854-af66-93ecd22bc3a4-kube-api-access-fghmj\") pod \"payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5\" (UID: \"bf02b0cb-f674-4854-af66-93ecd22bc3a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5"
Oct 30 22:38:00.697734 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:00.697706    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf02b0cb-f674-4854-af66-93ecd22bc3a4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5\" (UID: \"bf02b0cb-f674-4854-af66-93ecd22bc3a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5"
Oct 30 22:38:00.791030 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:00.790991    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:00.831295 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:00.824503    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fghmj\" (UniqueName: \"kubernetes.io/projected/bf02b0cb-f674-4854-af66-93ecd22bc3a4-kube-api-access-fghmj\") pod \"payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5\" (UID: \"bf02b0cb-f674-4854-af66-93ecd22bc3a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5"
Oct 30 22:38:00.831295 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:00.824699    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf02b0cb-f674-4854-af66-93ecd22bc3a4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5\" (UID: \"bf02b0cb-f674-4854-af66-93ecd22bc3a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5"
Oct 30 22:38:00.831295 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:00.825776    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf02b0cb-f674-4854-af66-93ecd22bc3a4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5\" (UID: \"bf02b0cb-f674-4854-af66-93ecd22bc3a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5"
Oct 30 22:38:01.053292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:01.048078    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fghmj\" (UniqueName: \"kubernetes.io/projected/bf02b0cb-f674-4854-af66-93ecd22bc3a4-kube-api-access-fghmj\") pod \"payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5\" (UID: \"bf02b0cb-f674-4854-af66-93ecd22bc3a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5"
Oct 30 22:38:01.224851 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:01.224758    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5"
Oct 30 22:38:01.547257 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:38:01.547184    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf02b0cb_f674_4854_af66_93ecd22bc3a4.slice/crio-3b33230477f85ecaa7997a4339c8a7899ae245c933f68bb663951813a8d09b0e WatchSource:0}: Error finding container 3b33230477f85ecaa7997a4339c8a7899ae245c933f68bb663951813a8d09b0e: Status 404 returned error can't find the container with id 3b33230477f85ecaa7997a4339c8a7899ae245c933f68bb663951813a8d09b0e
Oct 30 22:38:01.547832 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:01.547799    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5"]
Oct 30 22:38:01.587012 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:01.586983    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:01.762274 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:01.762165    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:01.928789 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:01.928685    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:02.063816 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:02.063780    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5" event={"ID":"bf02b0cb-f674-4854-af66-93ecd22bc3a4","Type":"ContainerStarted","Data":"8e23212e6789198052b2a0f5cfb8ddf88cb9561a96150a20ffc15f9d9e3623b9"}
Oct 30 22:38:02.064004 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:02.063864    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5" event={"ID":"bf02b0cb-f674-4854-af66-93ecd22bc3a4","Type":"ContainerStarted","Data":"3b33230477f85ecaa7997a4339c8a7899ae245c933f68bb663951813a8d09b0e"}
Oct 30 22:38:02.064004 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:02.063889    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5"
Oct 30 22:38:02.109767 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:02.109711    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5" podStartSLOduration=2.109698533 podStartE2EDuration="2.109698533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:02.106347597 +0000 UTC m=+1867.462714003" watchObservedRunningTime="2026-10-30 22:38:02.109698533 +0000 UTC m=+1867.466064848"
Oct 30 22:38:03.444249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:03.438828    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:05.551251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:05.547662    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:07.127558 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:07.127526    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:07.300371 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:07.300335    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:07.618506 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:07.617049    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:09.435674 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:09.435636    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:10.303736 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:38:10.303700    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:38:11.153249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:11.146583    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:12.953252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:12.946820    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:13.075774 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:13.075740    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5"
Oct 30 22:38:14.661191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:14.661161    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:16.418192 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:16.418150    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:16.581632 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:16.581599    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:16.749261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:16.749203    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:18.447135 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:18.447100    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:20.255415 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:20.255377    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:21.961957 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:21.961911    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:23.749826 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:23.749787    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:25.548251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:25.539165    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:27.357735 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:27.357700    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:29.330157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:29.330116    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:30.143924 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:30.143889    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:30.325001 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:30.324961    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:31.061774 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:31.060792    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:32.995249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:32.991901    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:34.846732 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:34.846690    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:36.656248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:36.651852    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:38.404804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:38.402752    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:38.920593 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:38.919615    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:39.119319 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:39.119273    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:39.294510 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:39.294473    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:39.489942 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:39.489905    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:40.269252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:40.264640    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:40.360451 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:38:40.360412    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:38:40.497247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:40.492154    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:40.858260 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:40.858208    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:42.662136 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:42.662084    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:43.745010 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:43.744966    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:43.919812 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:43.919771    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:44.158066 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:44.157976    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:44.759741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:44.759706    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:45.498615 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:45.498570    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:45.516692 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:45.516660    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv"]
Oct 30 22:38:45.523015 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:45.522993    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv"
Oct 30 22:38:45.540202 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:45.540145    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv"]
Oct 30 22:38:45.590787 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:45.590752    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ceb7603-9c15-4c92-9dbd-bd6c45828ef8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv\" (UID: \"8ceb7603-9c15-4c92-9dbd-bd6c45828ef8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv"
Oct 30 22:38:45.590983 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:45.590831    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd8cj\" (UniqueName: \"kubernetes.io/projected/8ceb7603-9c15-4c92-9dbd-bd6c45828ef8-kube-api-access-kd8cj\") pod \"payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv\" (UID: \"8ceb7603-9c15-4c92-9dbd-bd6c45828ef8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv"
Oct 30 22:38:45.658262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:45.654498    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-867f12\""
Oct 30 22:38:45.708032 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:45.707995    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ceb7603-9c15-4c92-9dbd-bd6c45828ef8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv\" (UID: \"8ceb7603-9c15-4c92-9dbd-bd6c45828ef8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv"
Oct 30 22:38:45.871321 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:45.870656    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd8cj\" (UniqueName: \"kubernetes.io/projected/8ceb7603-9c15-4c92-9dbd-bd6c45828ef8-kube-api-access-kd8cj\") pod \"payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv\" (UID: \"8ceb7603-9c15-4c92-9dbd-bd6c45828ef8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv"
Oct 30 22:38:45.871321 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:45.793714    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:45.871321 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:45.771957    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ceb7603-9c15-4c92-9dbd-bd6c45828ef8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv\" (UID: \"8ceb7603-9c15-4c92-9dbd-bd6c45828ef8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv"
Oct 30 22:38:45.923487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:45.922014    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd8cj\" (UniqueName: \"kubernetes.io/projected/8ceb7603-9c15-4c92-9dbd-bd6c45828ef8-kube-api-access-kd8cj\") pod \"payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv\" (UID: \"8ceb7603-9c15-4c92-9dbd-bd6c45828ef8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv"
Oct 30 22:38:46.062244 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:46.061323    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:46.141984 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:46.141898    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv"
Oct 30 22:38:46.555605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:46.555568    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv"]
Oct 30 22:38:46.555957 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:38:46.555916    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ceb7603_9c15_4c92_9dbd_bd6c45828ef8.slice/crio-6773f63d7c3a13aac86f464fa2354be67baa7d4a3d0034244f7e7c9b87ab0d38 WatchSource:0}: Error finding container 6773f63d7c3a13aac86f464fa2354be67baa7d4a3d0034244f7e7c9b87ab0d38: Status 404 returned error can't find the container with id 6773f63d7c3a13aac86f464fa2354be67baa7d4a3d0034244f7e7c9b87ab0d38
Oct 30 22:38:46.819162 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:46.819073    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv" event={"ID":"8ceb7603-9c15-4c92-9dbd-bd6c45828ef8","Type":"ContainerStarted","Data":"20a81b12ccb2708cb2029fbd1674c2150741ed20d5a46bbbc0b72705a9a9efd0"}
Oct 30 22:38:46.819162 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:46.819123    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv" event={"ID":"8ceb7603-9c15-4c92-9dbd-bd6c45828ef8","Type":"ContainerStarted","Data":"6773f63d7c3a13aac86f464fa2354be67baa7d4a3d0034244f7e7c9b87ab0d38"}
Oct 30 22:38:46.819162 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:46.819142    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv"
Oct 30 22:38:46.880707 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:46.880631    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv" podStartSLOduration=1.8806100940000001 podStartE2EDuration="1.880610094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:46.86350621 +0000 UTC m=+1912.219872527" watchObservedRunningTime="2026-10-30 22:38:46.880610094 +0000 UTC m=+1912.236976387"
Oct 30 22:38:47.081298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:47.074114    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:48.614173 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:48.614140    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:48.970823 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:48.970729    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:49.138424 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:49.138389    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:50.849752 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:50.849718    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:52.632255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:52.630363    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:52.802532 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:52.802496    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:54.423213 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:54.421951    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:56.485582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:56.485545    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:57.836102 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:57.836073    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv"
Oct 30 22:38:58.800279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:38:58.798952    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:00.448905 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:00.443895    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:00.822250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:00.812077    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:00.985030 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:00.984992    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:01.383398 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:01.383247    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:02.127578 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:02.127542    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:02.857257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:02.841083    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:04.735898 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:04.735859    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:05.392754 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:05.392720    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:05.560720 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:05.560685    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:06.594265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:06.588872    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:07.116150 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.116112    4889 generic.go:356] "Generic (PLEG): container finished" podID="2ced92c8-b2b3-4d29-86ca-b816484d7b69" containerID="1d5c8fa33d8ec6ac2d3999d9ce57aed5d988289a718cc4c15c5778a4cc654c5e" exitCode=0
Oct 30 22:39:07.116377 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.116184    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz" event={"ID":"2ced92c8-b2b3-4d29-86ca-b816484d7b69","Type":"ContainerDied","Data":"1d5c8fa33d8ec6ac2d3999d9ce57aed5d988289a718cc4c15c5778a4cc654c5e"}
Oct 30 22:39:07.116706 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.116692    4889 scope.go:122] "RemoveContainer" containerID="1d5c8fa33d8ec6ac2d3999d9ce57aed5d988289a718cc4c15c5778a4cc654c5e"
Oct 30 22:39:07.118852 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.118824    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:39:07.118988 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.118897    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:39:07.118988 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.118934    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:39:07.181678 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.181646    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:07.330445 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.330415    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf"]
Oct 30 22:39:07.335528 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.335503    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf"
Oct 30 22:39:07.353542 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.353483    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-3acfdb-dockercfg-cltkb\""
Oct 30 22:39:07.361446 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.361418    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf"]
Oct 30 22:39:07.367211 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.367126    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:07.367663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.367626    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-3acfdb\""
Oct 30 22:39:07.394042 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.394010    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5"]
Oct 30 22:39:07.537172 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.537122    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e006a58-a7ed-4f12-8c3d-f0db13e46fc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf\" (UID: \"8e006a58-a7ed-4f12-8c3d-f0db13e46fc2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf"
Oct 30 22:39:07.541801 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.541765    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2mwn\" (UniqueName: \"kubernetes.io/projected/8e006a58-a7ed-4f12-8c3d-f0db13e46fc2-kube-api-access-k2mwn\") pod \"payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf\" (UID: \"8e006a58-a7ed-4f12-8c3d-f0db13e46fc2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf"
Oct 30 22:39:07.677361 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.669899    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2mwn\" (UniqueName: \"kubernetes.io/projected/8e006a58-a7ed-4f12-8c3d-f0db13e46fc2-kube-api-access-k2mwn\") pod \"payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf\" (UID: \"8e006a58-a7ed-4f12-8c3d-f0db13e46fc2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf"
Oct 30 22:39:07.689514 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.678323    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e006a58-a7ed-4f12-8c3d-f0db13e46fc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf\" (UID: \"8e006a58-a7ed-4f12-8c3d-f0db13e46fc2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf"
Oct 30 22:39:07.708588 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.691047    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e006a58-a7ed-4f12-8c3d-f0db13e46fc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf\" (UID: \"8e006a58-a7ed-4f12-8c3d-f0db13e46fc2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf"
Oct 30 22:39:07.777577 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.777548    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5"
Oct 30 22:39:07.787350 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.786014    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5"]
Oct 30 22:39:07.796216 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.796183    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2mwn\" (UniqueName: \"kubernetes.io/projected/8e006a58-a7ed-4f12-8c3d-f0db13e46fc2-kube-api-access-k2mwn\") pod \"payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf\" (UID: \"8e006a58-a7ed-4f12-8c3d-f0db13e46fc2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf"
Oct 30 22:39:07.894068 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.894037    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twgj2\" (UniqueName: \"kubernetes.io/projected/92be936b-b994-4bb9-ba7c-53845b589684-kube-api-access-twgj2\") pod \"payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5\" (UID: \"92be936b-b994-4bb9-ba7c-53845b589684\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5"
Oct 30 22:39:07.894264 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.894160    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92be936b-b994-4bb9-ba7c-53845b589684-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5\" (UID: \"92be936b-b994-4bb9-ba7c-53845b589684\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5"
Oct 30 22:39:07.959272 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.959189    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf"
Oct 30 22:39:08.001242 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.996465    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twgj2\" (UniqueName: \"kubernetes.io/projected/92be936b-b994-4bb9-ba7c-53845b589684-kube-api-access-twgj2\") pod \"payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5\" (UID: \"92be936b-b994-4bb9-ba7c-53845b589684\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5"
Oct 30 22:39:08.001242 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.996567    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92be936b-b994-4bb9-ba7c-53845b589684-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5\" (UID: \"92be936b-b994-4bb9-ba7c-53845b589684\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5"
Oct 30 22:39:08.001242 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:07.997576    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92be936b-b994-4bb9-ba7c-53845b589684-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5\" (UID: \"92be936b-b994-4bb9-ba7c-53845b589684\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5"
Oct 30 22:39:08.018335 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:08.018300    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twgj2\" (UniqueName: \"kubernetes.io/projected/92be936b-b994-4bb9-ba7c-53845b589684-kube-api-access-twgj2\") pod \"payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5\" (UID: \"92be936b-b994-4bb9-ba7c-53845b589684\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5"
Oct 30 22:39:08.107981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:08.107942    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5"
Oct 30 22:39:08.166360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:08.166319    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz" event={"ID":"2ced92c8-b2b3-4d29-86ca-b816484d7b69","Type":"ContainerStarted","Data":"5a468b998c8581d9db3e8ccb6b740e8192310c7f102f9cc854b5d8c5162a39c2"}
Oct 30 22:39:08.302886 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:39:08.302853    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e006a58_a7ed_4f12_8c3d_f0db13e46fc2.slice/crio-b0517a05a4a74a90136ede69f86864da70a096b840673c14f69602908aed4091 WatchSource:0}: Error finding container b0517a05a4a74a90136ede69f86864da70a096b840673c14f69602908aed4091: Status 404 returned error can't find the container with id b0517a05a4a74a90136ede69f86864da70a096b840673c14f69602908aed4091
Oct 30 22:39:08.304205 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:08.304181    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf"]
Oct 30 22:39:08.461496 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:39:08.461467    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92be936b_b994_4bb9_ba7c_53845b589684.slice/crio-92130609c3e85283d6a438d434df289387a4b72c7b27e6945aac4daf388ff4db WatchSource:0}: Error finding container 92130609c3e85283d6a438d434df289387a4b72c7b27e6945aac4daf388ff4db: Status 404 returned error can't find the container with id 92130609c3e85283d6a438d434df289387a4b72c7b27e6945aac4daf388ff4db
Oct 30 22:39:08.462614 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:08.462593    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5"]
Oct 30 22:39:08.672146 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:08.669462    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:08.764279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:08.761782    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:39:08.764279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:08.762620    4889 patch_prober.go:29] interesting pod/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 22:39:08.764279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:08.762670    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz" podUID="2ced92c8-b2b3-4d29-86ca-b816484d7b69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 22:39:09.176389 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:09.176287    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf" event={"ID":"8e006a58-a7ed-4f12-8c3d-f0db13e46fc2","Type":"ContainerStarted","Data":"88ab58c202110a33de862dc5056bb841302a78e990ef7526e86ad348670cfd6f"}
Oct 30 22:39:09.176389 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:09.176353    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf"
Oct 30 22:39:09.176389 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:09.176375    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf" event={"ID":"8e006a58-a7ed-4f12-8c3d-f0db13e46fc2","Type":"ContainerStarted","Data":"b0517a05a4a74a90136ede69f86864da70a096b840673c14f69602908aed4091"}
Oct 30 22:39:09.178276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:09.178203    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5" event={"ID":"92be936b-b994-4bb9-ba7c-53845b589684","Type":"ContainerStarted","Data":"c455df5999a73be63a9464c1d9f1d4f878b85f931112c3662985a477e35eb913"}
Oct 30 22:39:09.178276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:09.178304    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5" event={"ID":"92be936b-b994-4bb9-ba7c-53845b589684","Type":"ContainerStarted","Data":"92130609c3e85283d6a438d434df289387a4b72c7b27e6945aac4daf388ff4db"}
Oct 30 22:39:09.178539 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:09.178427    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5"
Oct 30 22:39:09.238368 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:09.238208    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf" podStartSLOduration=2.2381935410000002 podStartE2EDuration="2.238193541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:09.233642034 +0000 UTC m=+1934.590008350" watchObservedRunningTime="2026-10-30 22:39:09.238193541 +0000 UTC m=+1934.594559857"
Oct 30 22:39:09.280199 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:09.280147    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5" podStartSLOduration=2.280134808 podStartE2EDuration="2.280134808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:09.276010414 +0000 UTC m=+1934.632376729" watchObservedRunningTime="2026-10-30 22:39:09.280134808 +0000 UTC m=+1934.636501146"
Oct 30 22:39:09.762143 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:09.762106    4889 patch_prober.go:29] interesting pod/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 22:39:09.762360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:09.762155    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz" podUID="2ced92c8-b2b3-4d29-86ca-b816484d7b69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 22:39:10.303423 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:39:10.303387    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:39:10.754255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:10.749920    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:10.772311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:10.765638    4889 patch_prober.go:29] interesting pod/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 22:39:10.772311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:10.765701    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz" podUID="2ced92c8-b2b3-4d29-86ca-b816484d7b69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 22:39:11.761741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:11.761706    4889 patch_prober.go:29] interesting pod/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 22:39:11.762209 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:11.761768    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz" podUID="2ced92c8-b2b3-4d29-86ca-b816484d7b69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 22:39:12.738250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:12.734870    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:12.774366 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:12.774295    4889 patch_prober.go:29] interesting pod/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 22:39:12.775076 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:12.775030    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz" podUID="2ced92c8-b2b3-4d29-86ca-b816484d7b69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 22:39:13.762272 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:13.761809    4889 patch_prober.go:29] interesting pod/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 22:39:13.762272 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:13.761855    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz" podUID="2ced92c8-b2b3-4d29-86ca-b816484d7b69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 22:39:14.743749 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:14.743711    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:14.803903 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:14.803873    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:39:14.804363 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:14.804342    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:39:14.805457 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:14.805436    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:39:14.936378 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:14.936341    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:16.556249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:16.556039    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:18.469881 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:18.462829    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:20.156285 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.156242    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:20.182007 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.181976    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"]
Oct 30 22:39:20.187514 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.187479    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"
Oct 30 22:39:20.202470 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.202444    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf"
Oct 30 22:39:20.206310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.206283    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"]
Oct 30 22:39:20.209881 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.209853    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5"
Oct 30 22:39:20.212154 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.212122    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-54c7f0\"/\"openshift-service-ca.crt\""
Oct 30 22:39:20.212154 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.212132    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-54c7f0\"/\"kube-root-ca.crt\""
Oct 30 22:39:20.220280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.220212    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-54c7f0\"/\"default-dockercfg-bcccn\""
Oct 30 22:39:20.220405 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.220353    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-54c7f0\"/\"test-model-54c7f0-kserve-self-signed-certs\""
Oct 30 22:39:20.306604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.306569    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b4e3337-ba72-4247-b521-85740c5a5c3a-tls-certs\") pod \"test-model-54c7f0-kserve-94c596b77-bjl5t\" (UID: \"0b4e3337-ba72-4247-b521-85740c5a5c3a\") " pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"
Oct 30 22:39:20.306802 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.306616    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-model-cache\") pod \"test-model-54c7f0-kserve-94c596b77-bjl5t\" (UID: \"0b4e3337-ba72-4247-b521-85740c5a5c3a\") " pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"
Oct 30 22:39:20.306802 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.306763    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8flst\" (UniqueName: \"kubernetes.io/projected/0b4e3337-ba72-4247-b521-85740c5a5c3a-kube-api-access-8flst\") pod \"test-model-54c7f0-kserve-94c596b77-bjl5t\" (UID: \"0b4e3337-ba72-4247-b521-85740c5a5c3a\") " pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"
Oct 30 22:39:20.306934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.306910    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-dshm\") pod \"test-model-54c7f0-kserve-94c596b77-bjl5t\" (UID: \"0b4e3337-ba72-4247-b521-85740c5a5c3a\") " pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"
Oct 30 22:39:20.307086 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.307034    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-tmp-dir\") pod \"test-model-54c7f0-kserve-94c596b77-bjl5t\" (UID: \"0b4e3337-ba72-4247-b521-85740c5a5c3a\") " pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"
Oct 30 22:39:20.307236 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.307207    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-home\") pod \"test-model-54c7f0-kserve-94c596b77-bjl5t\" (UID: \"0b4e3337-ba72-4247-b521-85740c5a5c3a\") " pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"
Oct 30 22:39:20.353682 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.353640    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:20.408883 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.408793    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-dshm\") pod \"test-model-54c7f0-kserve-94c596b77-bjl5t\" (UID: \"0b4e3337-ba72-4247-b521-85740c5a5c3a\") " pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"
Oct 30 22:39:20.408883 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.408876    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-tmp-dir\") pod \"test-model-54c7f0-kserve-94c596b77-bjl5t\" (UID: \"0b4e3337-ba72-4247-b521-85740c5a5c3a\") " pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"
Oct 30 22:39:20.409118 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.408957    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-home\") pod \"test-model-54c7f0-kserve-94c596b77-bjl5t\" (UID: \"0b4e3337-ba72-4247-b521-85740c5a5c3a\") " pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"
Oct 30 22:39:20.409118 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.409014    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b4e3337-ba72-4247-b521-85740c5a5c3a-tls-certs\") pod \"test-model-54c7f0-kserve-94c596b77-bjl5t\" (UID: \"0b4e3337-ba72-4247-b521-85740c5a5c3a\") " pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"
Oct 30 22:39:20.409279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.409154    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-model-cache\") pod \"test-model-54c7f0-kserve-94c596b77-bjl5t\" (UID: \"0b4e3337-ba72-4247-b521-85740c5a5c3a\") " pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"
Oct 30 22:39:20.409344 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.409307    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-tmp-dir\") pod \"test-model-54c7f0-kserve-94c596b77-bjl5t\" (UID: \"0b4e3337-ba72-4247-b521-85740c5a5c3a\") " pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"
Oct 30 22:39:20.409457 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.409314    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8flst\" (UniqueName: \"kubernetes.io/projected/0b4e3337-ba72-4247-b521-85740c5a5c3a-kube-api-access-8flst\") pod \"test-model-54c7f0-kserve-94c596b77-bjl5t\" (UID: \"0b4e3337-ba72-4247-b521-85740c5a5c3a\") " pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"
Oct 30 22:39:20.409536 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.409345    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-home\") pod \"test-model-54c7f0-kserve-94c596b77-bjl5t\" (UID: \"0b4e3337-ba72-4247-b521-85740c5a5c3a\") " pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"
Oct 30 22:39:20.409595 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.409573    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-model-cache\") pod \"test-model-54c7f0-kserve-94c596b77-bjl5t\" (UID: \"0b4e3337-ba72-4247-b521-85740c5a5c3a\") " pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"
Oct 30 22:39:20.411270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.411213    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-dshm\") pod \"test-model-54c7f0-kserve-94c596b77-bjl5t\" (UID: \"0b4e3337-ba72-4247-b521-85740c5a5c3a\") " pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"
Oct 30 22:39:20.411610 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.411593    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b4e3337-ba72-4247-b521-85740c5a5c3a-tls-certs\") pod \"test-model-54c7f0-kserve-94c596b77-bjl5t\" (UID: \"0b4e3337-ba72-4247-b521-85740c5a5c3a\") " pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"
Oct 30 22:39:20.430496 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.430474    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8flst\" (UniqueName: \"kubernetes.io/projected/0b4e3337-ba72-4247-b521-85740c5a5c3a-kube-api-access-8flst\") pod \"test-model-54c7f0-kserve-94c596b77-bjl5t\" (UID: \"0b4e3337-ba72-4247-b521-85740c5a5c3a\") " pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"
Oct 30 22:39:20.526249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.517781    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"
Oct 30 22:39:20.966025 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:39:20.965990    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b4e3337_ba72_4247_b521_85740c5a5c3a.slice/crio-12b8bafa30aefb55dbd134d74abd5a0579d011b05493740828528187e2d3e0b3 WatchSource:0}: Error finding container 12b8bafa30aefb55dbd134d74abd5a0579d011b05493740828528187e2d3e0b3: Status 404 returned error can't find the container with id 12b8bafa30aefb55dbd134d74abd5a0579d011b05493740828528187e2d3e0b3
Oct 30 22:39:20.967660 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:20.967636    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"]
Oct 30 22:39:21.310898 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:21.310863    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t" event={"ID":"0b4e3337-ba72-4247-b521-85740c5a5c3a","Type":"ContainerStarted","Data":"17c6c74500939f16ab15280c9df6a86374198bfbce877381cb4f1268cb22a832"}
Oct 30 22:39:21.311380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:21.310907    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t" event={"ID":"0b4e3337-ba72-4247-b521-85740c5a5c3a","Type":"ContainerStarted","Data":"12b8bafa30aefb55dbd134d74abd5a0579d011b05493740828528187e2d3e0b3"}
Oct 30 22:39:21.358002 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:21.357945    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t" podStartSLOduration=1.357928633 podStartE2EDuration="1.357928633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:21.353622979 +0000 UTC m=+1946.709989333" watchObservedRunningTime="2026-10-30 22:39:21.357928633 +0000 UTC m=+1946.714294964"
Oct 30 22:39:21.524386 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:21.518272    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"
Oct 30 22:39:21.970786 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:21.963111    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:23.725250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:23.718948    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:25.442716 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:25.442678    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:26.176563 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.176530    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"]
Oct 30 22:39:26.182002 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.181978    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"
Oct 30 22:39:26.203271 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.203206    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"]
Oct 30 22:39:26.210082 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.210056    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d85da4\"/\"kube-root-ca.crt\""
Oct 30 22:39:26.210305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.210141    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d85da4\"/\"openshift-service-ca.crt\""
Oct 30 22:39:26.219191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.219152    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d85da4\"/\"test-model-d85da4-kserve-self-signed-certs\""
Oct 30 22:39:26.219355 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.219274    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d85da4\"/\"default-dockercfg-k5rqp\""
Oct 30 22:39:26.251461 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.251422    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b0d1da0-5e38-4000-826a-01b8110db1a1-tls-certs\") pod \"test-model-d85da4-kserve-64556965c9-4zh6w\" (UID: \"3b0d1da0-5e38-4000-826a-01b8110db1a1\") " pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"
Oct 30 22:39:26.251652 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.251527    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-home\") pod \"test-model-d85da4-kserve-64556965c9-4zh6w\" (UID: \"3b0d1da0-5e38-4000-826a-01b8110db1a1\") " pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"
Oct 30 22:39:26.251652 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.251578    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn5bv\" (UniqueName: \"kubernetes.io/projected/3b0d1da0-5e38-4000-826a-01b8110db1a1-kube-api-access-cn5bv\") pod \"test-model-d85da4-kserve-64556965c9-4zh6w\" (UID: \"3b0d1da0-5e38-4000-826a-01b8110db1a1\") " pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"
Oct 30 22:39:26.251963 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.251933    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-tmp-dir\") pod \"test-model-d85da4-kserve-64556965c9-4zh6w\" (UID: \"3b0d1da0-5e38-4000-826a-01b8110db1a1\") " pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"
Oct 30 22:39:26.252104 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.252054    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-dshm\") pod \"test-model-d85da4-kserve-64556965c9-4zh6w\" (UID: \"3b0d1da0-5e38-4000-826a-01b8110db1a1\") " pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"
Oct 30 22:39:26.252203 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.252187    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-model-cache\") pod \"test-model-d85da4-kserve-64556965c9-4zh6w\" (UID: \"3b0d1da0-5e38-4000-826a-01b8110db1a1\") " pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"
Oct 30 22:39:26.305044 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.305007    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:26.353264 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.353202    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-dshm\") pod \"test-model-d85da4-kserve-64556965c9-4zh6w\" (UID: \"3b0d1da0-5e38-4000-826a-01b8110db1a1\") " pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"
Oct 30 22:39:26.353442 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.353326    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-model-cache\") pod \"test-model-d85da4-kserve-64556965c9-4zh6w\" (UID: \"3b0d1da0-5e38-4000-826a-01b8110db1a1\") " pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"
Oct 30 22:39:26.353442 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.353439    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b0d1da0-5e38-4000-826a-01b8110db1a1-tls-certs\") pod \"test-model-d85da4-kserve-64556965c9-4zh6w\" (UID: \"3b0d1da0-5e38-4000-826a-01b8110db1a1\") " pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"
Oct 30 22:39:26.353580 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.353511    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-home\") pod \"test-model-d85da4-kserve-64556965c9-4zh6w\" (UID: \"3b0d1da0-5e38-4000-826a-01b8110db1a1\") " pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"
Oct 30 22:39:26.353580 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.353559    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn5bv\" (UniqueName: \"kubernetes.io/projected/3b0d1da0-5e38-4000-826a-01b8110db1a1-kube-api-access-cn5bv\") pod \"test-model-d85da4-kserve-64556965c9-4zh6w\" (UID: \"3b0d1da0-5e38-4000-826a-01b8110db1a1\") " pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"
Oct 30 22:39:26.353685 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.353615    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-tmp-dir\") pod \"test-model-d85da4-kserve-64556965c9-4zh6w\" (UID: \"3b0d1da0-5e38-4000-826a-01b8110db1a1\") " pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"
Oct 30 22:39:26.353805 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.353781    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-model-cache\") pod \"test-model-d85da4-kserve-64556965c9-4zh6w\" (UID: \"3b0d1da0-5e38-4000-826a-01b8110db1a1\") " pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"
Oct 30 22:39:26.353903 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.353872    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-home\") pod \"test-model-d85da4-kserve-64556965c9-4zh6w\" (UID: \"3b0d1da0-5e38-4000-826a-01b8110db1a1\") " pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"
Oct 30 22:39:26.354072 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.354051    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-tmp-dir\") pod \"test-model-d85da4-kserve-64556965c9-4zh6w\" (UID: \"3b0d1da0-5e38-4000-826a-01b8110db1a1\") " pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"
Oct 30 22:39:26.356187 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.356159    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-dshm\") pod \"test-model-d85da4-kserve-64556965c9-4zh6w\" (UID: \"3b0d1da0-5e38-4000-826a-01b8110db1a1\") " pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"
Oct 30 22:39:26.356651 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.356620    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b0d1da0-5e38-4000-826a-01b8110db1a1-tls-certs\") pod \"test-model-d85da4-kserve-64556965c9-4zh6w\" (UID: \"3b0d1da0-5e38-4000-826a-01b8110db1a1\") " pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"
Oct 30 22:39:26.375046 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.375026    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn5bv\" (UniqueName: \"kubernetes.io/projected/3b0d1da0-5e38-4000-826a-01b8110db1a1-kube-api-access-cn5bv\") pod \"test-model-d85da4-kserve-64556965c9-4zh6w\" (UID: \"3b0d1da0-5e38-4000-826a-01b8110db1a1\") " pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"
Oct 30 22:39:26.513262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.504552    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"
Oct 30 22:39:26.909820 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:26.909785    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"]
Oct 30 22:39:26.911515 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:39:26.911487    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b0d1da0_5e38_4000_826a_01b8110db1a1.slice/crio-28cd27f98459d9df05e33ed63a1f81760d8326ddcc9407e859a055f7da6e6755 WatchSource:0}: Error finding container 28cd27f98459d9df05e33ed63a1f81760d8326ddcc9407e859a055f7da6e6755: Status 404 returned error can't find the container with id 28cd27f98459d9df05e33ed63a1f81760d8326ddcc9407e859a055f7da6e6755
Oct 30 22:39:27.367751 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:27.367710    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w" event={"ID":"3b0d1da0-5e38-4000-826a-01b8110db1a1","Type":"ContainerStarted","Data":"7390b7477a2a96d2e10b883e9e674df80f6a2290f46babd82660be5f9fb93edc"}
Oct 30 22:39:27.367954 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:27.367761    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w" event={"ID":"3b0d1da0-5e38-4000-826a-01b8110db1a1","Type":"ContainerStarted","Data":"28cd27f98459d9df05e33ed63a1f81760d8326ddcc9407e859a055f7da6e6755"}
Oct 30 22:39:27.426299 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:27.424101    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w" podStartSLOduration=1.42408176 podStartE2EDuration="1.42408176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:27.417875131 +0000 UTC m=+1952.774241448" watchObservedRunningTime="2026-10-30 22:39:27.42408176 +0000 UTC m=+1952.780448077"
Oct 30 22:39:27.428642 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:27.428575    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:27.505527 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:27.505496    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"
Oct 30 22:39:27.626348 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:27.626059    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:29.450273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:29.444331    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:30.208503 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:30.208466    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:30.518467 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:30.518436    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"
Oct 30 22:39:30.532928 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:30.532898    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"
Oct 30 22:39:31.469177 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:31.469148    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"
Oct 30 22:39:31.640544 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:31.640423    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:33.455481 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:33.455448    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:35.292251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:35.287963    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:36.505005 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:36.504974    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"
Oct 30 22:39:36.518372 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:36.518345    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"
Oct 30 22:39:36.530187 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:36.530166    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"
Oct 30 22:39:37.079270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:37.073677    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:37.356566 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:37.356478    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:39.129983 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:39.129882    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:40.304401 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:39:40.304362    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:39:41.012635 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:41.012555    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:42.703354 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:42.703269    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:44.450602 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:44.450544    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:46.298328 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:46.298292    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:48.181250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:48.178941    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:48.731751 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:48.731713    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:50.130250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:50.128864    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:51.538768 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:51.538729    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:52.077456 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:52.077408    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:53.872876 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:53.872841    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:54.097524 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:54.097489    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:55.641734 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:55.641695    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:57.465988 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:57.456035    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:59.157951 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:59.157908    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:59.342529 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:59.342491    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:59.601954 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:39:59.601919    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:01.143773 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:01.143735    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:02.965778 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:02.965724    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:04.282141 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:04.282104    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:04.949019 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:04.948983    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:06.998859 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:06.998826    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:08.954794 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:08.954751    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:10.304047 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:40:10.304013    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:40:10.786257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:10.780866    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:12.561261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:12.558885    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:14.277581 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:14.277543    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:16.064414 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:16.064380    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:17.879084 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:17.879051    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:19.641241 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:19.641181    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:21.468423 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:21.468351    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:23.166307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:23.166174    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:24.977380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:24.977340    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:26.740793 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:26.740759    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:28.425057 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:28.425019    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:30.134022 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:30.133983    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:31.699682 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:31.699643    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:31.937151 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:31.937056    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:33.740635 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:33.740580    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:35.636329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:35.635949    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:36.343549 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:36.343514    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:37.443089 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:37.442513    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:39.220002 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:39.219964    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:40.304263 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:40:40.304187    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:40:41.030876 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:41.030838    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:42.685375 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:42.685333    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"]
Oct 30 22:40:42.689696 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:42.689667    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:42.697087 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:42.697059    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"
Oct 30 22:40:42.707535 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:42.707508    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"]
Oct 30 22:40:42.715616 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:42.715586    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bc48cc\"/\"openshift-service-ca.crt\""
Oct 30 22:40:42.728798 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:42.728768    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bc48cc\"/\"kube-root-ca.crt\""
Oct 30 22:40:42.735935 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:42.729313    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bc48cc\"/\"test-model-bc48cc-kserve-self-signed-certs\""
Oct 30 22:40:42.736101 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:42.729392    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bc48cc\"/\"default-dockercfg-hbksz\""
Oct 30 22:40:42.757996 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:42.757967    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-home\") pod \"test-model-bc48cc-kserve-599cd96549-wfl49\" (UID: \"31fba7f5-e583-4c1f-af77-01d86ec53b57\") " pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"
Oct 30 22:40:42.758117 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:42.758085    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-dshm\") pod \"test-model-bc48cc-kserve-599cd96549-wfl49\" (UID: \"31fba7f5-e583-4c1f-af77-01d86ec53b57\") " pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"
Oct 30 22:40:42.758166 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:42.758131    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-tmp-dir\") pod \"test-model-bc48cc-kserve-599cd96549-wfl49\" (UID: \"31fba7f5-e583-4c1f-af77-01d86ec53b57\") " pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"
Oct 30 22:40:42.758203 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:42.758186    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31fba7f5-e583-4c1f-af77-01d86ec53b57-tls-certs\") pod \"test-model-bc48cc-kserve-599cd96549-wfl49\" (UID: \"31fba7f5-e583-4c1f-af77-01d86ec53b57\") " pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"
Oct 30 22:40:42.758341 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:42.758314    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-model-cache\") pod \"test-model-bc48cc-kserve-599cd96549-wfl49\" (UID: \"31fba7f5-e583-4c1f-af77-01d86ec53b57\") " pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"
Oct 30 22:40:42.758388 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:42.758376    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdpgf\" (UniqueName: \"kubernetes.io/projected/31fba7f5-e583-4c1f-af77-01d86ec53b57-kube-api-access-sdpgf\") pod \"test-model-bc48cc-kserve-599cd96549-wfl49\" (UID: \"31fba7f5-e583-4c1f-af77-01d86ec53b57\") " pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"
Oct 30 22:40:42.859309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:42.859279    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-dshm\") pod \"test-model-bc48cc-kserve-599cd96549-wfl49\" (UID: \"31fba7f5-e583-4c1f-af77-01d86ec53b57\") " pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"
Oct 30 22:40:42.859492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:42.859331    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-tmp-dir\") pod \"test-model-bc48cc-kserve-599cd96549-wfl49\" (UID: \"31fba7f5-e583-4c1f-af77-01d86ec53b57\") " pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"
Oct 30 22:40:42.859492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:42.859405    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31fba7f5-e583-4c1f-af77-01d86ec53b57-tls-certs\") pod \"test-model-bc48cc-kserve-599cd96549-wfl49\" (UID: \"31fba7f5-e583-4c1f-af77-01d86ec53b57\") " pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"
Oct 30 22:40:42.859695 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:42.859562    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-model-cache\") pod \"test-model-bc48cc-kserve-599cd96549-wfl49\" (UID: \"31fba7f5-e583-4c1f-af77-01d86ec53b57\") " pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"
Oct 30 22:40:42.859695 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:42.859621    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdpgf\" (UniqueName: \"kubernetes.io/projected/31fba7f5-e583-4c1f-af77-01d86ec53b57-kube-api-access-sdpgf\") pod \"test-model-bc48cc-kserve-599cd96549-wfl49\" (UID: \"31fba7f5-e583-4c1f-af77-01d86ec53b57\") " pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"
Oct 30 22:40:42.859828 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:42.859793    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-tmp-dir\") pod \"test-model-bc48cc-kserve-599cd96549-wfl49\" (UID: \"31fba7f5-e583-4c1f-af77-01d86ec53b57\") " pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"
Oct 30 22:40:42.859944 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:42.859917    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-home\") pod \"test-model-bc48cc-kserve-599cd96549-wfl49\" (UID: \"31fba7f5-e583-4c1f-af77-01d86ec53b57\") " pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"
Oct 30 22:40:42.860117 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:42.859926    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-model-cache\") pod \"test-model-bc48cc-kserve-599cd96549-wfl49\" (UID: \"31fba7f5-e583-4c1f-af77-01d86ec53b57\") " pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"
Oct 30 22:40:42.860327 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:42.860309    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-home\") pod \"test-model-bc48cc-kserve-599cd96549-wfl49\" (UID: \"31fba7f5-e583-4c1f-af77-01d86ec53b57\") " pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"
Oct 30 22:40:42.862052 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:42.862026    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-dshm\") pod \"test-model-bc48cc-kserve-599cd96549-wfl49\" (UID: \"31fba7f5-e583-4c1f-af77-01d86ec53b57\") " pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"
Oct 30 22:40:42.862179 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:42.862163    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31fba7f5-e583-4c1f-af77-01d86ec53b57-tls-certs\") pod \"test-model-bc48cc-kserve-599cd96549-wfl49\" (UID: \"31fba7f5-e583-4c1f-af77-01d86ec53b57\") " pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"
Oct 30 22:40:42.870483 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:42.870460    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:42.880425 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:42.880402    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdpgf\" (UniqueName: \"kubernetes.io/projected/31fba7f5-e583-4c1f-af77-01d86ec53b57-kube-api-access-sdpgf\") pod \"test-model-bc48cc-kserve-599cd96549-wfl49\" (UID: \"31fba7f5-e583-4c1f-af77-01d86ec53b57\") " pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"
Oct 30 22:40:43.026848 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:43.026519    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"
Oct 30 22:40:43.409623 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:40:43.409591    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31fba7f5_e583_4c1f_af77_01d86ec53b57.slice/crio-a59a7e55a87989473e5688eeab2eb2a69292ac206cfac1dcfb2ac03f37e3d7fc WatchSource:0}: Error finding container a59a7e55a87989473e5688eeab2eb2a69292ac206cfac1dcfb2ac03f37e3d7fc: Status 404 returned error can't find the container with id a59a7e55a87989473e5688eeab2eb2a69292ac206cfac1dcfb2ac03f37e3d7fc
Oct 30 22:40:43.411236 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:43.411202    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"]
Oct 30 22:40:44.374628 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:44.374592    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49" event={"ID":"31fba7f5-e583-4c1f-af77-01d86ec53b57","Type":"ContainerStarted","Data":"5bd41712edcea971bb34b98f8e5cb1ae528e3f75fda5d7ca0dc9ef990a4f771f"}
Oct 30 22:40:44.374628 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:44.374637    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49" event={"ID":"31fba7f5-e583-4c1f-af77-01d86ec53b57","Type":"ContainerStarted","Data":"a59a7e55a87989473e5688eeab2eb2a69292ac206cfac1dcfb2ac03f37e3d7fc"}
Oct 30 22:40:44.423619 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:44.423550    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49" podStartSLOduration=2.423528993 podStartE2EDuration="2.423528993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:44.422533005 +0000 UTC m=+2029.778899319" watchObservedRunningTime="2026-10-30 22:40:44.423528993 +0000 UTC m=+2029.779895361"
Oct 30 22:40:44.470900 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:44.470860    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:45.028010 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:45.027973    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"
Oct 30 22:40:46.138566 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:46.138529    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:47.827649 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:47.827612    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:48.854574 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:48.854539    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:49.531182 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:49.531137    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:51.244048 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:51.244003    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:52.993797 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:52.990798    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:53.030619 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:53.030588    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"
Oct 30 22:40:53.048555 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:53.048524    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"
Oct 30 22:40:53.459594 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:53.459512    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"
Oct 30 22:40:54.945206 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:54.945169    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:55.943690 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:55.943656    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"]
Oct 30 22:40:55.943890 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:55.943854    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79" podUID="9759df3a-c1a8-40e6-a087-dfd96b243c12" containerName="istio-proxy" containerID="cri-o://0bb85fba1e2d8a07e00919e24b53f97a98f42dce95c5d98d0962914e15e7404e" gracePeriod=30
Oct 30 22:40:56.856445 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:56.856409    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:57.976169 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:57.976008    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:58.624256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:40:58.623615    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:00.430291 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:00.430152    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:00.756690 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:00.756641    4889 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:41:00.756859 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:00.756716    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79" podUID="9759df3a-c1a8-40e6-a087-dfd96b243c12" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:41:01.249183 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.249155    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:41:01.351768 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.351673    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-data\") pod \"9759df3a-c1a8-40e6-a087-dfd96b243c12\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") "
Oct 30 22:41:01.351951 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.351801    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-envoy\") pod \"9759df3a-c1a8-40e6-a087-dfd96b243c12\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") "
Oct 30 22:41:01.351951 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.351861    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-podinfo\") pod \"9759df3a-c1a8-40e6-a087-dfd96b243c12\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") "
Oct 30 22:41:01.352086 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.351961    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-data" pod "9759df3a-c1a8-40e6-a087-dfd96b243c12" (UID: "9759df3a-c1a8-40e6-a087-dfd96b243c12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:01.352086 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.352032    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9759df3a-c1a8-40e6-a087-dfd96b243c12-kube-api-access-kgn2m\" (UniqueName: \"kubernetes.io/projected/9759df3a-c1a8-40e6-a087-dfd96b243c12-kube-api-access-kgn2m\") pod \"9759df3a-c1a8-40e6-a087-dfd96b243c12\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") "
Oct 30 22:41:01.352086 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.352077    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-token\" (UniqueName: \"kubernetes.io/projected/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-token\") pod \"9759df3a-c1a8-40e6-a087-dfd96b243c12\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") "
Oct 30 22:41:01.352913 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.352175    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-credential-socket\") pod \"9759df3a-c1a8-40e6-a087-dfd96b243c12\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") "
Oct 30 22:41:01.352913 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.352270    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-workload-socket\") pod \"9759df3a-c1a8-40e6-a087-dfd96b243c12\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") "
Oct 30 22:41:01.352913 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.352303    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9759df3a-c1a8-40e6-a087-dfd96b243c12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9759df3a-c1a8-40e6-a087-dfd96b243c12-istiod-ca-cert\") pod \"9759df3a-c1a8-40e6-a087-dfd96b243c12\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") "
Oct 30 22:41:01.352913 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.352347    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-workload-certs\") pod \"9759df3a-c1a8-40e6-a087-dfd96b243c12\" (UID: \"9759df3a-c1a8-40e6-a087-dfd96b243c12\") "
Oct 30 22:41:01.353342 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.353301    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-workload-socket" pod "9759df3a-c1a8-40e6-a087-dfd96b243c12" (UID: "9759df3a-c1a8-40e6-a087-dfd96b243c12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:01.353474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.353385    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-credential-socket" pod "9759df3a-c1a8-40e6-a087-dfd96b243c12" (UID: "9759df3a-c1a8-40e6-a087-dfd96b243c12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:01.353663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.353633    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-workload-certs" pod "9759df3a-c1a8-40e6-a087-dfd96b243c12" (UID: "9759df3a-c1a8-40e6-a087-dfd96b243c12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:01.353990 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.353970    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9759df3a-c1a8-40e6-a087-dfd96b243c12-istiod-ca-cert" pod "9759df3a-c1a8-40e6-a087-dfd96b243c12" (UID: "9759df3a-c1a8-40e6-a087-dfd96b243c12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:01.355937 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.354942    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:41:01.355937 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.354970    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:41:01.355937 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.354986    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9759df3a-c1a8-40e6-a087-dfd96b243c12-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:41:01.355937 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.354998    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:41:01.355937 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.355010    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:41:01.355937 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.355089    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-podinfo" pod "9759df3a-c1a8-40e6-a087-dfd96b243c12" (UID: "9759df3a-c1a8-40e6-a087-dfd96b243c12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:41:01.355937 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.355620    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-token" pod "9759df3a-c1a8-40e6-a087-dfd96b243c12" (UID: "9759df3a-c1a8-40e6-a087-dfd96b243c12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:01.355937 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.355785    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-envoy" pod "9759df3a-c1a8-40e6-a087-dfd96b243c12" (UID: "9759df3a-c1a8-40e6-a087-dfd96b243c12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:01.356493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.356319    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9759df3a-c1a8-40e6-a087-dfd96b243c12-kube-api-access-kgn2m" pod "9759df3a-c1a8-40e6-a087-dfd96b243c12" (UID: "9759df3a-c1a8-40e6-a087-dfd96b243c12"). InnerVolumeSpecName "kube-api-access-kgn2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:01.456834 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.456799    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:41:01.456834 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.456835    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:41:01.457323 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.456850    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgn2m\" (UniqueName: \"kubernetes.io/projected/9759df3a-c1a8-40e6-a087-dfd96b243c12-kube-api-access-kgn2m\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:41:01.457323 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.456866    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9759df3a-c1a8-40e6-a087-dfd96b243c12-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:41:01.596373 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.596334    4889 generic.go:356] "Generic (PLEG): container finished" podID="9759df3a-c1a8-40e6-a087-dfd96b243c12" containerID="0bb85fba1e2d8a07e00919e24b53f97a98f42dce95c5d98d0962914e15e7404e" exitCode=0
Oct 30 22:41:01.596575 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.596395    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"
Oct 30 22:41:01.596676 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.596636    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79" event={"ID":"9759df3a-c1a8-40e6-a087-dfd96b243c12","Type":"ContainerDied","Data":"0bb85fba1e2d8a07e00919e24b53f97a98f42dce95c5d98d0962914e15e7404e"}
Oct 30 22:41:01.596750 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.596698    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79" event={"ID":"9759df3a-c1a8-40e6-a087-dfd96b243c12","Type":"ContainerDied","Data":"0f54fef16b4955831ca35a23293cf17655b512ba1cd7d4174ec9f9d5ade433ad"}
Oct 30 22:41:01.596750 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.596723    4889 scope.go:122] "RemoveContainer" containerID="0bb85fba1e2d8a07e00919e24b53f97a98f42dce95c5d98d0962914e15e7404e"
Oct 30 22:41:01.612336 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.612311    4889 scope.go:122] "RemoveContainer" containerID="0bb85fba1e2d8a07e00919e24b53f97a98f42dce95c5d98d0962914e15e7404e"
Oct 30 22:41:01.612693 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:41:01.612671    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bb85fba1e2d8a07e00919e24b53f97a98f42dce95c5d98d0962914e15e7404e\": container with ID starting with 0bb85fba1e2d8a07e00919e24b53f97a98f42dce95c5d98d0962914e15e7404e not found: ID does not exist" containerID="0bb85fba1e2d8a07e00919e24b53f97a98f42dce95c5d98d0962914e15e7404e"
Oct 30 22:41:01.612778 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.612703    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bb85fba1e2d8a07e00919e24b53f97a98f42dce95c5d98d0962914e15e7404e"} err="failed to get container status \"0bb85fba1e2d8a07e00919e24b53f97a98f42dce95c5d98d0962914e15e7404e\": rpc error: code = NotFound desc = could not find container \"0bb85fba1e2d8a07e00919e24b53f97a98f42dce95c5d98d0962914e15e7404e\": container with ID starting with 0bb85fba1e2d8a07e00919e24b53f97a98f42dce95c5d98d0962914e15e7404e not found: ID does not exist"
Oct 30 22:41:01.660980 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.660943    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"]
Oct 30 22:41:01.666808 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:01.666773    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qtv79"]
Oct 30 22:41:02.365632 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:02.361604    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:02.778530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:02.778496    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9759df3a-c1a8-40e6-a087-dfd96b243c12" path="/var/lib/kubelet/pods/9759df3a-c1a8-40e6-a087-dfd96b243c12/volumes"
Oct 30 22:41:04.075430 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:04.075001    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:05.835251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:05.828027    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:07.718195 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:07.713533    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:09.418056 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:09.418010    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:10.305311 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:41:10.305274    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:41:11.327248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:11.325999    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:13.584488 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:13.584449    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:13.813121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:13.813082    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:15.561892 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:15.561856    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:17.413113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:17.410486    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:19.268941 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:19.268902    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:19.806107 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:19.806067    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"]
Oct 30 22:41:19.806967 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:19.806950    4889 cpu_manager.go:414] "containerMap: removing container" podUID="9759df3a-c1a8-40e6-a087-dfd96b243c12" containerName="istio-proxy"
Oct 30 22:41:19.807020 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:19.806970    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9759df3a-c1a8-40e6-a087-dfd96b243c12" containerName="istio-proxy"
Oct 30 22:41:19.807203 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:19.807189    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="9759df3a-c1a8-40e6-a087-dfd96b243c12" containerName="istio-proxy"
Oct 30 22:41:19.812297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:19.811447    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"
Oct 30 22:41:19.832211 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:19.832179    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"]
Oct 30 22:41:19.840998 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:19.840962    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6e0edc\"/\"default-dockercfg-6hcvn\""
Oct 30 22:41:19.840998 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:19.840985    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6e0edc\"/\"test-model-6e0edc-kserve-self-signed-certs\""
Oct 30 22:41:19.848446 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:19.848419    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6e0edc\"/\"openshift-service-ca.crt\""
Oct 30 22:41:19.848655 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:19.848441    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6e0edc\"/\"kube-root-ca.crt\""
Oct 30 22:41:19.904383 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:19.904334    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-dshm\") pod \"test-model-6e0edc-kserve-67b89f554f-tbjc4\" (UID: \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\") " pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"
Oct 30 22:41:19.904562 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:19.904505    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-model-cache\") pod \"test-model-6e0edc-kserve-67b89f554f-tbjc4\" (UID: \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\") " pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"
Oct 30 22:41:19.904562 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:19.904535    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-tmp-dir\") pod \"test-model-6e0edc-kserve-67b89f554f-tbjc4\" (UID: \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\") " pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"
Oct 30 22:41:19.904680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:19.904661    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-home\") pod \"test-model-6e0edc-kserve-67b89f554f-tbjc4\" (UID: \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\") " pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"
Oct 30 22:41:19.904788 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:19.904768    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzl66\" (UniqueName: \"kubernetes.io/projected/e7acdf19-b537-4edf-91f8-8b95a03a1f43-kube-api-access-dzl66\") pod \"test-model-6e0edc-kserve-67b89f554f-tbjc4\" (UID: \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\") " pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"
Oct 30 22:41:19.904852 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:19.904841    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7acdf19-b537-4edf-91f8-8b95a03a1f43-tls-certs\") pod \"test-model-6e0edc-kserve-67b89f554f-tbjc4\" (UID: \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\") " pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"
Oct 30 22:41:19.944835 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:19.944766    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:20.006603 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:20.006567    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-home\") pod \"test-model-6e0edc-kserve-67b89f554f-tbjc4\" (UID: \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\") " pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"
Oct 30 22:41:20.006799 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:20.006656    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzl66\" (UniqueName: \"kubernetes.io/projected/e7acdf19-b537-4edf-91f8-8b95a03a1f43-kube-api-access-dzl66\") pod \"test-model-6e0edc-kserve-67b89f554f-tbjc4\" (UID: \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\") " pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"
Oct 30 22:41:20.006799 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:20.006717    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7acdf19-b537-4edf-91f8-8b95a03a1f43-tls-certs\") pod \"test-model-6e0edc-kserve-67b89f554f-tbjc4\" (UID: \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\") " pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"
Oct 30 22:41:20.006799 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:20.006793    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-dshm\") pod \"test-model-6e0edc-kserve-67b89f554f-tbjc4\" (UID: \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\") " pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"
Oct 30 22:41:20.006978 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:20.006893    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-model-cache\") pod \"test-model-6e0edc-kserve-67b89f554f-tbjc4\" (UID: \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\") " pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"
Oct 30 22:41:20.006978 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:20.006927    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-tmp-dir\") pod \"test-model-6e0edc-kserve-67b89f554f-tbjc4\" (UID: \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\") " pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"
Oct 30 22:41:20.007102 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:20.007015    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-home\") pod \"test-model-6e0edc-kserve-67b89f554f-tbjc4\" (UID: \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\") " pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"
Oct 30 22:41:20.007277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:20.007212    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-model-cache\") pod \"test-model-6e0edc-kserve-67b89f554f-tbjc4\" (UID: \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\") " pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"
Oct 30 22:41:20.007381 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:20.007313    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-tmp-dir\") pod \"test-model-6e0edc-kserve-67b89f554f-tbjc4\" (UID: \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\") " pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"
Oct 30 22:41:20.009274 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:20.009249    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-dshm\") pod \"test-model-6e0edc-kserve-67b89f554f-tbjc4\" (UID: \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\") " pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"
Oct 30 22:41:20.009538 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:20.009523    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7acdf19-b537-4edf-91f8-8b95a03a1f43-tls-certs\") pod \"test-model-6e0edc-kserve-67b89f554f-tbjc4\" (UID: \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\") " pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"
Oct 30 22:41:20.029545 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:20.029519    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzl66\" (UniqueName: \"kubernetes.io/projected/e7acdf19-b537-4edf-91f8-8b95a03a1f43-kube-api-access-dzl66\") pod \"test-model-6e0edc-kserve-67b89f554f-tbjc4\" (UID: \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\") " pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"
Oct 30 22:41:20.147322 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:20.144558    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"
Oct 30 22:41:20.549794 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:41:20.549763    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7acdf19_b537_4edf_91f8_8b95a03a1f43.slice/crio-31612fe552ea87c7f0dafceb2aab4225921b2bd15de9e57e74fd4e9b531e9e9e WatchSource:0}: Error finding container 31612fe552ea87c7f0dafceb2aab4225921b2bd15de9e57e74fd4e9b531e9e9e: Status 404 returned error can't find the container with id 31612fe552ea87c7f0dafceb2aab4225921b2bd15de9e57e74fd4e9b531e9e9e
Oct 30 22:41:20.551718 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:20.551693    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"]
Oct 30 22:41:20.892915 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:20.892820    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4" event={"ID":"e7acdf19-b537-4edf-91f8-8b95a03a1f43","Type":"ContainerStarted","Data":"7cffd4b4e65fc11cb5e4f35d643ba7be36a4986601e5cda56eca2876be5adfac"}
Oct 30 22:41:20.892915 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:20.892867    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4" event={"ID":"e7acdf19-b537-4edf-91f8-8b95a03a1f43","Type":"ContainerStarted","Data":"31612fe552ea87c7f0dafceb2aab4225921b2bd15de9e57e74fd4e9b531e9e9e"}
Oct 30 22:41:20.940537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:20.940486    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4" podStartSLOduration=1.940470537 podStartE2EDuration="1.940470537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:20.936458974 +0000 UTC m=+2066.292825288" watchObservedRunningTime="2026-10-30 22:41:20.940470537 +0000 UTC m=+2066.296836872"
Oct 30 22:41:21.145659 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:21.145579    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"
Oct 30 22:41:21.358254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:21.357828    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:23.138436 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:23.138340    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:25.574733 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.574696    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"]
Oct 30 22:41:25.580046 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.580026    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.602191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.602160    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"]
Oct 30 22:41:25.608743 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.608718    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c13db0-openshift-default-dockercfg-ppd9k\""
Oct 30 22:41:25.630773 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.630743    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnhgn\" (UniqueName: \"kubernetes.io/projected/78af4d09-3007-46d3-9b8e-043a4f536c26-kube-api-access-mnhgn\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.630922 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.630782    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78af4d09-3007-46d3-9b8e-043a4f536c26-istiod-ca-cert\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.630984 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.630960    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-workload-certs\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.631052 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.631041    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-token\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.631122 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.631103    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-workload-socket\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.631160 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.631142    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-data\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.631270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.631203    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-podinfo\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.631504 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.631479    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-credential-socket\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.631578 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.631525    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-envoy\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.733532 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.733495    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-podinfo\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.733705 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.733567    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-credential-socket\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.733705 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.733600    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-envoy\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.733850 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.733712    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnhgn\" (UniqueName: \"kubernetes.io/projected/78af4d09-3007-46d3-9b8e-043a4f536c26-kube-api-access-mnhgn\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.733850 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.733742    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78af4d09-3007-46d3-9b8e-043a4f536c26-istiod-ca-cert\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.733963 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.733901    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-workload-certs\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.734021 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.734004    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-token\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.734078 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.734031    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-credential-socket\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.734131 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.734107    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-workload-socket\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.734192 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.734139    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-data\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.734192 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.734181    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-workload-certs\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.734540 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.734517    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-workload-socket\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.734540 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.734531    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-data\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.734762 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.734741    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78af4d09-3007-46d3-9b8e-043a4f536c26-istiod-ca-cert\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.736135 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.736113    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-envoy\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.736281 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.736262    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-podinfo\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.757685 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.757657    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-token\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.757826 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.757806    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnhgn\" (UniqueName: \"kubernetes.io/projected/78af4d09-3007-46d3-9b8e-043a4f536c26-kube-api-access-mnhgn\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.897219 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.897124    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:25.916960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:25.916931    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:26.230025 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:41:26.229994    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78af4d09_3007_46d3_9b8e_043a4f536c26.slice/crio-3fb148465e6906fee6236f65c5324fbc74dda8b16f53b231d1515f3b0748bb33 WatchSource:0}: Error finding container 3fb148465e6906fee6236f65c5324fbc74dda8b16f53b231d1515f3b0748bb33: Status 404 returned error can't find the container with id 3fb148465e6906fee6236f65c5324fbc74dda8b16f53b231d1515f3b0748bb33
Oct 30 22:41:26.231329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:26.231306    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"]
Oct 30 22:41:26.233413 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:26.233382    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:41:26.233497 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:26.233448    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:41:26.233497 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:26.233476    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:41:26.832253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:26.823691    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:26.980074 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:26.980039    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c" event={"ID":"78af4d09-3007-46d3-9b8e-043a4f536c26","Type":"ContainerStarted","Data":"fd40f2adc8766c131ef7b56112904978b9e06f409d0205cda5e3f88b2cf4cc29"}
Oct 30 22:41:26.980292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:26.980104    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c" event={"ID":"78af4d09-3007-46d3-9b8e-043a4f536c26","Type":"ContainerStarted","Data":"3fb148465e6906fee6236f65c5324fbc74dda8b16f53b231d1515f3b0748bb33"}
Oct 30 22:41:27.063968 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:27.063919    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c" podStartSLOduration=2.063905541 podStartE2EDuration="2.063905541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:27.047629683 +0000 UTC m=+2072.403996011" watchObservedRunningTime="2026-10-30 22:41:27.063905541 +0000 UTC m=+2072.420271921"
Oct 30 22:41:27.897552 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:27.897519    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:27.899773 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:27.899749    4889 patch_prober.go:29] interesting pod/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 30 22:41:27.899910 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:27.899794    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c" podUID="78af4d09-3007-46d3-9b8e-043a4f536c26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 30 22:41:28.763259 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:28.756917    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:28.898160 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:28.898118    4889 patch_prober.go:29] interesting pod/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 30 22:41:28.898639 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:28.898187    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c" podUID="78af4d09-3007-46d3-9b8e-043a4f536c26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 30 22:41:29.898001 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:29.897964    4889 patch_prober.go:29] interesting pod/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 30 22:41:29.898194 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:29.898027    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c" podUID="78af4d09-3007-46d3-9b8e-043a4f536c26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 30 22:41:30.145304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:30.145268    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"
Oct 30 22:41:30.158660 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:30.158572    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"
Oct 30 22:41:30.626899 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:30.626850    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:30.898408 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:30.898306    4889 patch_prober.go:29] interesting pod/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 30 22:41:30.898408 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:30.898389    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c" podUID="78af4d09-3007-46d3-9b8e-043a4f536c26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 30 22:41:31.062934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:31.062898    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"
Oct 30 22:41:31.898006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:31.897975    4889 patch_prober.go:29] interesting pod/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 30 22:41:31.898192 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:31.898024    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c" podUID="78af4d09-3007-46d3-9b8e-043a4f536c26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 30 22:41:32.580042 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:32.580008    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:32.897607 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:32.897516    4889 patch_prober.go:29] interesting pod/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 30 22:41:32.897607 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:32.897596    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c" podUID="78af4d09-3007-46d3-9b8e-043a4f536c26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 30 22:41:33.904294 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:33.904261    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:33.904785 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:33.904758    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:33.905464 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:33.905442    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:41:34.447020 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:34.446980    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:36.236367 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:36.236314    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:38.100261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:38.095539    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:38.791753 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:38.791714    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"]
Oct 30 22:41:38.797526 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:38.797496    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:38.820436 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:38.820404    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"]
Oct 30 22:41:38.827927 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:38.827891    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f0bb45-openshift-default-dockercfg-f5d62\""
Oct 30 22:41:38.902373 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:38.902339    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-workload-certs\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:38.902574 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:38.902400    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-workload-socket\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:38.902574 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:38.902539    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-envoy\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:38.902574 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:38.902561    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-token\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:38.902726 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:38.902586    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-data\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:38.902726 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:38.902607    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istiod-ca-cert\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:38.902809 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:38.902777    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-credential-socket\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:38.902861 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:38.902835    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-podinfo\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:38.902965 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:38.902941    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8rmv\" (UniqueName: \"kubernetes.io/projected/19ea5d7a-f7db-40f2-ad56-66e429ea7393-kube-api-access-z8rmv\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:39.004570 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:39.004534    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-credential-socket\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:39.004810 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:39.004608    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-podinfo\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:39.004810 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:39.004663    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8rmv\" (UniqueName: \"kubernetes.io/projected/19ea5d7a-f7db-40f2-ad56-66e429ea7393-kube-api-access-z8rmv\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:39.004810 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:39.004733    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-workload-certs\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:39.004810 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:39.004796    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-workload-socket\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:39.005086 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:39.004910    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-envoy\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:39.005086 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:39.004942    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-token\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:39.005086 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:39.004982    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-data\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:39.005086 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:39.005015    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istiod-ca-cert\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:39.005086 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:39.005071    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-workload-certs\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:39.005385 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:39.005011    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-credential-socket\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:39.005385 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:39.005238    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-workload-socket\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:39.005548 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:39.005518    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-data\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:39.005822 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:39.005804    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istiod-ca-cert\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:39.007489 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:39.007462    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-envoy\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:39.007615 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:39.007598    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-podinfo\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:39.031943 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:39.031900    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-token\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:39.032113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:39.032090    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8rmv\" (UniqueName: \"kubernetes.io/projected/19ea5d7a-f7db-40f2-ad56-66e429ea7393-kube-api-access-z8rmv\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:39.117688 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:39.117604    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:39.448264 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:41:39.448203    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ea5d7a_f7db_40f2_ad56_66e429ea7393.slice/crio-a3bbacb841d9848828b0226d5c477215e4832f732969f3c7774a47ed47fbbc9b WatchSource:0}: Error finding container a3bbacb841d9848828b0226d5c477215e4832f732969f3c7774a47ed47fbbc9b: Status 404 returned error can't find the container with id a3bbacb841d9848828b0226d5c477215e4832f732969f3c7774a47ed47fbbc9b
Oct 30 22:41:39.449903 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:39.449879    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"]
Oct 30 22:41:39.452283 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:39.452248    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:41:39.452361 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:39.452328    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:41:39.452401 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:39.452372    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:41:39.860590 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:39.860540    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:40.057303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:40.056481    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:40.204479 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:40.202880    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4" event={"ID":"19ea5d7a-f7db-40f2-ad56-66e429ea7393","Type":"ContainerStarted","Data":"9261a037fe2a36e8d8bcb499ad03cf04c47319124f7e1600acd02ef43b9044e9"}
Oct 30 22:41:40.204479 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:40.202967    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4" event={"ID":"19ea5d7a-f7db-40f2-ad56-66e429ea7393","Type":"ContainerStarted","Data":"a3bbacb841d9848828b0226d5c477215e4832f732969f3c7774a47ed47fbbc9b"}
Oct 30 22:41:40.282200 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:40.282138    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4" podStartSLOduration=2.282121229 podStartE2EDuration="2.282121229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:40.267070662 +0000 UTC m=+2085.623436980" watchObservedRunningTime="2026-10-30 22:41:40.282121229 +0000 UTC m=+2085.638487543"
Oct 30 22:41:40.311552 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:41:40.306906    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:41:41.124251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:41.118647    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:41.136655 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:41.136604    4889 patch_prober.go:29] interesting pod/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 30 22:41:41.136869 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:41.136677    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4" podUID="19ea5d7a-f7db-40f2-ad56-66e429ea7393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 30 22:41:41.971252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:41.965514    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:42.130250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:42.129723    4889 patch_prober.go:29] interesting pod/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 30 22:41:42.130250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:42.129790    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4" podUID="19ea5d7a-f7db-40f2-ad56-66e429ea7393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 30 22:41:43.131250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:43.125632    4889 patch_prober.go:29] interesting pod/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 30 22:41:43.131250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:43.125698    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4" podUID="19ea5d7a-f7db-40f2-ad56-66e429ea7393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 30 22:41:43.784784 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:43.784743    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:44.134499 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:44.134403    4889 patch_prober.go:29] interesting pod/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 30 22:41:44.134499 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:44.134470    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4" podUID="19ea5d7a-f7db-40f2-ad56-66e429ea7393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 30 22:41:45.130249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:45.129664    4889 patch_prober.go:29] interesting pod/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 30 22:41:45.130249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:45.129738    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4" podUID="19ea5d7a-f7db-40f2-ad56-66e429ea7393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 30 22:41:45.664148 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:45.664100    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:46.118637 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:46.118591    4889 patch_prober.go:29] interesting pod/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 30 22:41:46.118871 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:46.118644    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4" podUID="19ea5d7a-f7db-40f2-ad56-66e429ea7393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 30 22:41:47.124058 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:47.124028    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:47.124514 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:47.124491    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:47.125201 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:47.125184    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:41:47.468729 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:47.468693    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:48.968542 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:48.968508    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:49.237850 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:49.237761    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:50.931151 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:50.931116    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:52.127259 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:52.106484    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:52.821511 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:52.821472    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:54.648249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:54.644406    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:55.550680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:55.550648    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 22:41:55.558912 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:55.558884    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 22:41:56.468447 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:56.468351    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:58.146657 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:58.144557    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:59.868325 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:41:59.868190    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:01.604032 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:01.603992    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:03.288260 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:03.287031    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:05.144675 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:05.144409    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:06.907849 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:06.907806    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:08.749631 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:08.749595    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:10.305553 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:42:10.305507    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:42:10.626034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:10.625878    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:12.533855 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:12.533819    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:14.316022 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:14.310937    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:15.285284 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:15.285024    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:15.697552 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:15.697452    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:15.866734 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:15.866698    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:16.276344 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:16.275252    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:18.462250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:18.458681    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:20.549578 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:20.549539    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:22.389279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:22.389168    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:24.255975 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:24.255917    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:25.303481 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:25.303445    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"]
Oct 30 22:42:25.303923 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:25.303680    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv" podUID="246eac1b-e89f-4433-930d-f8adbd60524f" containerName="istio-proxy" containerID="cri-o://e860586c0e376bb1fd31759e996d438d196fe3c6e27296b100e7ce4ff813beac" gracePeriod=30
Oct 30 22:42:25.320293 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:25.320211    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"]
Oct 30 22:42:25.320562 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:25.320528    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln" podUID="31604bd8-b5a2-435f-b021-f4f29b31b8ce" containerName="istio-proxy" containerID="cri-o://bcd65b23813851dffffe16db781dc6ecf5012a5e1e917f91abbe0e7c508121f8" gracePeriod=30
Oct 30 22:42:25.773386 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:25.773352    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"]
Oct 30 22:42:25.778897 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:25.778872    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:25.798124 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:25.798081    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"]
Oct 30 22:42:25.804820 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:25.804788    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-867f12-openshift-default-dockercfg-tfg6j\""
Oct 30 22:42:25.897433 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:25.897390    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-podinfo\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:25.897627 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:25.897463    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e842f91-5282-4f41-b149-d8ce4c8625a5-istiod-ca-cert\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:25.897680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:25.897651    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-envoy\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:25.897730 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:25.897691    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l67dj\" (UniqueName: \"kubernetes.io/projected/6e842f91-5282-4f41-b149-d8ce4c8625a5-kube-api-access-l67dj\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:25.897830 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:25.897811    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-data\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:25.897897 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:25.897865    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-credential-socket\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:25.897960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:25.897908    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-workload-socket\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:25.898017 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:25.898007    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-workload-certs\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:25.898146 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:25.898132    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-token\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:26.000005 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:25.999962    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-podinfo\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:26.000005 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.000013    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e842f91-5282-4f41-b149-d8ce4c8625a5-istiod-ca-cert\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:26.000299 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.000271    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-envoy\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:26.000350 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.000324    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l67dj\" (UniqueName: \"kubernetes.io/projected/6e842f91-5282-4f41-b149-d8ce4c8625a5-kube-api-access-l67dj\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:26.000418 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.000400    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-data\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:26.000475 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.000451    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-credential-socket\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:26.000532 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.000490    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-workload-socket\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:26.000673 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.000632    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-workload-certs\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:26.000857 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.000837    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-token\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:26.001027 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.001006    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e842f91-5282-4f41-b149-d8ce4c8625a5-istiod-ca-cert\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:26.001594 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.001562    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-data\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:26.001594 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.001589    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-workload-socket\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:26.002115 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.001614    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-credential-socket\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:26.002115 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.001683    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-workload-certs\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:26.003268 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.003200    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-podinfo\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:26.003874 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.003856    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-envoy\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:26.024047 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.023967    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-token\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:26.024316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.024291    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l67dj\" (UniqueName: \"kubernetes.io/projected/6e842f91-5282-4f41-b149-d8ce4c8625a5-kube-api-access-l67dj\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:26.031427 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.031395    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq"]
Oct 30 22:42:26.069825 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.069770    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:26.100611 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.100563    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:26.130896 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.130830    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"]
Oct 30 22:42:26.131288 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.131135    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg" podUID="7221740b-e003-4af1-86a2-396e32fff7ca" containerName="istio-proxy" containerID="cri-o://b9bef27d10252e835d86a70b9b33765d31305448b82624b38a8434991f107d55" gracePeriod=30
Oct 30 22:42:26.251135 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.251081    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:26.527841 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:42:26.527811    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e842f91_5282_4f41_b149_d8ce4c8625a5.slice/crio-c925739945a5ddf8fce4d02edaacf0c64dfaeb18f7fa5df950d1ab6961460b0c WatchSource:0}: Error finding container c925739945a5ddf8fce4d02edaacf0c64dfaeb18f7fa5df950d1ab6961460b0c: Status 404 returned error can't find the container with id c925739945a5ddf8fce4d02edaacf0c64dfaeb18f7fa5df950d1ab6961460b0c
Oct 30 22:42:26.528196 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.528024    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"]
Oct 30 22:42:26.531409 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.531374    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:42:26.531543 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.531458    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:42:26.531543 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.531506    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:42:26.569069 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.569035    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"]
Oct 30 22:42:26.569286 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.569263    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt" podUID="137329f9-9437-46ab-b030-ead038ccd542" containerName="istio-proxy" containerID="cri-o://c49ea9e2f2ae9fc1a105e6a0647ad8743c92428eea6417eeb109bb0a6837ba3c" gracePeriod=30
Oct 30 22:42:26.905514 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.905390    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv" event={"ID":"6e842f91-5282-4f41-b149-d8ce4c8625a5","Type":"ContainerStarted","Data":"4863c9013c0a4784f630af6862219f193752101b94d196ab5c9daf94f22cb2a5"}
Oct 30 22:42:26.905885 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.905859    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv" event={"ID":"6e842f91-5282-4f41-b149-d8ce4c8625a5","Type":"ContainerStarted","Data":"c925739945a5ddf8fce4d02edaacf0c64dfaeb18f7fa5df950d1ab6961460b0c"}
Oct 30 22:42:26.979330 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:26.977033    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv" podStartSLOduration=1.9770131640000002 podStartE2EDuration="1.977013164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:26.958929011 +0000 UTC m=+2132.315295328" watchObservedRunningTime="2026-10-30 22:42:26.977013164 +0000 UTC m=+2132.333379481"
Oct 30 22:42:27.101499 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:27.101454    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:27.102955 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:27.102922    4889 patch_prober.go:29] interesting pod/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 22:42:27.103128 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:27.102976    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv" podUID="6e842f91-5282-4f41-b149-d8ce4c8625a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 22:42:27.725866 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:27.725828    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:28.127245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:28.124742    4889 patch_prober.go:29] interesting pod/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 22:42:28.127245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:28.124815    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv" podUID="6e842f91-5282-4f41-b149-d8ce4c8625a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 22:42:28.496119 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:28.496079    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:28.693513 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:28.693463    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:28.704393 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:28.704353    4889 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:42:28.704603 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:28.704434    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt" podUID="137329f9-9437-46ab-b030-ead038ccd542" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:42:28.931663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:28.931575    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:29.101853 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:29.101814    4889 patch_prober.go:29] interesting pod/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 22:42:29.102016 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:29.101887    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv" podUID="6e842f91-5282-4f41-b149-d8ce4c8625a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 22:42:29.236567 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:29.236462    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:29.957932 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:29.957892    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cmklv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:42:29.958403 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:29.957958    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv" podUID="246eac1b-e89f-4433-930d-f8adbd60524f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:42:30.101742 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.101702    4889 patch_prober.go:29] interesting pod/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 22:42:30.101932 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.101760    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv" podUID="6e842f91-5282-4f41-b149-d8ce4c8625a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 22:42:30.750466 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.750436    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:42:30.859211 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.859135    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:42:30.866388 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.866349    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/246eac1b-e89f-4433-930d-f8adbd60524f-istio-token\" (UniqueName: \"kubernetes.io/projected/246eac1b-e89f-4433-930d-f8adbd60524f-istio-token\") pod \"246eac1b-e89f-4433-930d-f8adbd60524f\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") "
Oct 30 22:42:30.866559 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.866426    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-istio-data\") pod \"246eac1b-e89f-4433-930d-f8adbd60524f\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") "
Oct 30 22:42:30.866559 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.866475    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/246eac1b-e89f-4433-930d-f8adbd60524f-kube-api-access-z5ls5\" (UniqueName: \"kubernetes.io/projected/246eac1b-e89f-4433-930d-f8adbd60524f-kube-api-access-z5ls5\") pod \"246eac1b-e89f-4433-930d-f8adbd60524f\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") "
Oct 30 22:42:30.866645 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.866599    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/246eac1b-e89f-4433-930d-f8adbd60524f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246eac1b-e89f-4433-930d-f8adbd60524f-istiod-ca-cert\") pod \"246eac1b-e89f-4433-930d-f8adbd60524f\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") "
Oct 30 22:42:30.866772 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.866753    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-credential-socket\") pod \"246eac1b-e89f-4433-930d-f8adbd60524f\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") "
Oct 30 22:42:30.866855 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.866794    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-workload-certs\") pod \"246eac1b-e89f-4433-930d-f8adbd60524f\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") "
Oct 30 22:42:30.866923 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.866851    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/246eac1b-e89f-4433-930d-f8adbd60524f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246eac1b-e89f-4433-930d-f8adbd60524f-istio-podinfo\") pod \"246eac1b-e89f-4433-930d-f8adbd60524f\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") "
Oct 30 22:42:30.866923 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.866878    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-istio-envoy\") pod \"246eac1b-e89f-4433-930d-f8adbd60524f\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") "
Oct 30 22:42:30.867046 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.866963    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-istio-data" pod "246eac1b-e89f-4433-930d-f8adbd60524f" (UID: "246eac1b-e89f-4433-930d-f8adbd60524f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:30.867107 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.867058    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/246eac1b-e89f-4433-930d-f8adbd60524f-istiod-ca-cert" pod "246eac1b-e89f-4433-930d-f8adbd60524f" (UID: "246eac1b-e89f-4433-930d-f8adbd60524f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:30.867161 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.867127    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-workload-socket\") pod \"246eac1b-e89f-4433-930d-f8adbd60524f\" (UID: \"246eac1b-e89f-4433-930d-f8adbd60524f\") "
Oct 30 22:42:30.867241 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.867205    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-workload-certs" pod "246eac1b-e89f-4433-930d-f8adbd60524f" (UID: "246eac1b-e89f-4433-930d-f8adbd60524f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:30.867318 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.867277    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-credential-socket" pod "246eac1b-e89f-4433-930d-f8adbd60524f" (UID: "246eac1b-e89f-4433-930d-f8adbd60524f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:30.867919 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.867891    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-workload-socket" pod "246eac1b-e89f-4433-930d-f8adbd60524f" (UID: "246eac1b-e89f-4433-930d-f8adbd60524f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:30.868121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.868100    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:30.868196 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.868128    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:30.868196 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.868142    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:30.868196 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.868157    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:30.868196 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.868170    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246eac1b-e89f-4433-930d-f8adbd60524f-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:30.869395 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.869367    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246eac1b-e89f-4433-930d-f8adbd60524f-istio-token" pod "246eac1b-e89f-4433-930d-f8adbd60524f" (UID: "246eac1b-e89f-4433-930d-f8adbd60524f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:30.870826 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.870785    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/246eac1b-e89f-4433-930d-f8adbd60524f-istio-podinfo" pod "246eac1b-e89f-4433-930d-f8adbd60524f" (UID: "246eac1b-e89f-4433-930d-f8adbd60524f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:42:30.870826 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.870801    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246eac1b-e89f-4433-930d-f8adbd60524f-kube-api-access-z5ls5" pod "246eac1b-e89f-4433-930d-f8adbd60524f" (UID: "246eac1b-e89f-4433-930d-f8adbd60524f"). InnerVolumeSpecName "kube-api-access-z5ls5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:30.871005 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.870962    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-istio-envoy" pod "246eac1b-e89f-4433-930d-f8adbd60524f" (UID: "246eac1b-e89f-4433-930d-f8adbd60524f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:30.969924 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.969301    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31604bd8-b5a2-435f-b021-f4f29b31b8ce-kube-api-access-b5vd9\" (UniqueName: \"kubernetes.io/projected/31604bd8-b5a2-435f-b021-f4f29b31b8ce-kube-api-access-b5vd9\") pod \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") "
Oct 30 22:42:30.969924 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.969386    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-workload-certs\") pod \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") "
Oct 30 22:42:30.969924 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.969496    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-data\") pod \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") "
Oct 30 22:42:30.969924 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.969607    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-envoy\") pod \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") "
Oct 30 22:42:30.969924 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.969686    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istiod-ca-cert\") pod \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") "
Oct 30 22:42:30.969924 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.969712    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-token\" (UniqueName: \"kubernetes.io/projected/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-token\") pod \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") "
Oct 30 22:42:30.969924 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.969843    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-credential-socket\") pod \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") "
Oct 30 22:42:30.970796 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.970010    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-podinfo\") pod \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") "
Oct 30 22:42:30.970796 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.970074    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-workload-socket\") pod \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\" (UID: \"31604bd8-b5a2-435f-b021-f4f29b31b8ce\") "
Oct 30 22:42:30.970796 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.970119    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istiod-ca-cert" pod "31604bd8-b5a2-435f-b021-f4f29b31b8ce" (UID: "31604bd8-b5a2-435f-b021-f4f29b31b8ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:30.970796 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.970443    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-credential-socket" pod "31604bd8-b5a2-435f-b021-f4f29b31b8ce" (UID: "31604bd8-b5a2-435f-b021-f4f29b31b8ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:30.970796 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.970615    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-workload-socket" pod "31604bd8-b5a2-435f-b021-f4f29b31b8ce" (UID: "31604bd8-b5a2-435f-b021-f4f29b31b8ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:30.971069 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.970911    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5ls5\" (UniqueName: \"kubernetes.io/projected/246eac1b-e89f-4433-930d-f8adbd60524f-kube-api-access-z5ls5\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:30.971069 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.970923    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246eac1b-e89f-4433-930d-f8adbd60524f-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:30.971069 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.970933    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246eac1b-e89f-4433-930d-f8adbd60524f-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:30.971069 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.970942    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:30.971069 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.970950    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:30.971069 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.970959    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:30.971069 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.970968    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246eac1b-e89f-4433-930d-f8adbd60524f-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:30.971540 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.971499    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-workload-certs" pod "31604bd8-b5a2-435f-b021-f4f29b31b8ce" (UID: "31604bd8-b5a2-435f-b021-f4f29b31b8ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:30.972693 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.972396    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-data" pod "31604bd8-b5a2-435f-b021-f4f29b31b8ce" (UID: "31604bd8-b5a2-435f-b021-f4f29b31b8ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:30.974692 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.974657    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-token" pod "31604bd8-b5a2-435f-b021-f4f29b31b8ce" (UID: "31604bd8-b5a2-435f-b021-f4f29b31b8ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:30.975060 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.975031    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-envoy" pod "31604bd8-b5a2-435f-b021-f4f29b31b8ce" (UID: "31604bd8-b5a2-435f-b021-f4f29b31b8ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:30.975359 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.975340    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-podinfo" pod "31604bd8-b5a2-435f-b021-f4f29b31b8ce" (UID: "31604bd8-b5a2-435f-b021-f4f29b31b8ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:42:30.975717 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:30.975688    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31604bd8-b5a2-435f-b021-f4f29b31b8ce-kube-api-access-b5vd9" pod "31604bd8-b5a2-435f-b021-f4f29b31b8ce" (UID: "31604bd8-b5a2-435f-b021-f4f29b31b8ce"). InnerVolumeSpecName "kube-api-access-b5vd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:31.036374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.031833    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq" podUID="c24e6bd5-4b2f-4060-b4cb-b2df03309b3a" containerName="payload-pre-processing" containerID="cri-o://75dccf023d12f60a21d74c5e905a437b983724c19129e41b7229fd55cf5b1e48" gracePeriod=25
Oct 30 22:42:31.088261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.086113    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:31.088261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.086157    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:31.088261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.086175    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5vd9\" (UniqueName: \"kubernetes.io/projected/31604bd8-b5a2-435f-b021-f4f29b31b8ce-kube-api-access-b5vd9\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:31.088261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.086190    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:31.088261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.086203    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:31.088261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.086215    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31604bd8-b5a2-435f-b021-f4f29b31b8ce-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:31.109652 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.103584    4889 patch_prober.go:29] interesting pod/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 22:42:31.109652 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.103648    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv" podUID="6e842f91-5282-4f41-b149-d8ce4c8625a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 22:42:31.133254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.118449    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:31.203006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.202970    4889 generic.go:356] "Generic (PLEG): container finished" podID="246eac1b-e89f-4433-930d-f8adbd60524f" containerID="e860586c0e376bb1fd31759e996d438d196fe3c6e27296b100e7ce4ff813beac" exitCode=0
Oct 30 22:42:31.203166 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.203023    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"
Oct 30 22:42:31.203166 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.203057    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv" event={"ID":"246eac1b-e89f-4433-930d-f8adbd60524f","Type":"ContainerDied","Data":"e860586c0e376bb1fd31759e996d438d196fe3c6e27296b100e7ce4ff813beac"}
Oct 30 22:42:31.203166 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.203117    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv" event={"ID":"246eac1b-e89f-4433-930d-f8adbd60524f","Type":"ContainerDied","Data":"94522c7c6942013fce77c769aaa987890051fb5f4c6c0c7045776068c194380f"}
Oct 30 22:42:31.203166 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.203147    4889 scope.go:122] "RemoveContainer" containerID="e860586c0e376bb1fd31759e996d438d196fe3c6e27296b100e7ce4ff813beac"
Oct 30 22:42:31.204935 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.204905    4889 generic.go:356] "Generic (PLEG): container finished" podID="31604bd8-b5a2-435f-b021-f4f29b31b8ce" containerID="bcd65b23813851dffffe16db781dc6ecf5012a5e1e917f91abbe0e7c508121f8" exitCode=0
Oct 30 22:42:31.205174 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.204958    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"
Oct 30 22:42:31.205174 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.205049    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln" event={"ID":"31604bd8-b5a2-435f-b021-f4f29b31b8ce","Type":"ContainerDied","Data":"bcd65b23813851dffffe16db781dc6ecf5012a5e1e917f91abbe0e7c508121f8"}
Oct 30 22:42:31.205174 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.205096    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln" event={"ID":"31604bd8-b5a2-435f-b021-f4f29b31b8ce","Type":"ContainerDied","Data":"1cc63f80ac88f6ce35e880faa84cc51e213c0cd18b173e153ab89f244e205840"}
Oct 30 22:42:31.210041 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.209710    4889 generic.go:356] "Generic (PLEG): container finished" podID="7221740b-e003-4af1-86a2-396e32fff7ca" containerID="b9bef27d10252e835d86a70b9b33765d31305448b82624b38a8434991f107d55" exitCode=0
Oct 30 22:42:31.210041 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.209936    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg" event={"ID":"7221740b-e003-4af1-86a2-396e32fff7ca","Type":"ContainerDied","Data":"b9bef27d10252e835d86a70b9b33765d31305448b82624b38a8434991f107d55"}
Oct 30 22:42:31.245008 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.244962    4889 scope.go:122] "RemoveContainer" containerID="e860586c0e376bb1fd31759e996d438d196fe3c6e27296b100e7ce4ff813beac"
Oct 30 22:42:31.245613 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:42:31.245585    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e860586c0e376bb1fd31759e996d438d196fe3c6e27296b100e7ce4ff813beac\": container with ID starting with e860586c0e376bb1fd31759e996d438d196fe3c6e27296b100e7ce4ff813beac not found: ID does not exist" containerID="e860586c0e376bb1fd31759e996d438d196fe3c6e27296b100e7ce4ff813beac"
Oct 30 22:42:31.245781 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.245747    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e860586c0e376bb1fd31759e996d438d196fe3c6e27296b100e7ce4ff813beac"} err="failed to get container status \"e860586c0e376bb1fd31759e996d438d196fe3c6e27296b100e7ce4ff813beac\": rpc error: code = NotFound desc = could not find container \"e860586c0e376bb1fd31759e996d438d196fe3c6e27296b100e7ce4ff813beac\": container with ID starting with e860586c0e376bb1fd31759e996d438d196fe3c6e27296b100e7ce4ff813beac not found: ID does not exist"
Oct 30 22:42:31.245937 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.245782    4889 scope.go:122] "RemoveContainer" containerID="bcd65b23813851dffffe16db781dc6ecf5012a5e1e917f91abbe0e7c508121f8"
Oct 30 22:42:31.259366 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.259340    4889 scope.go:122] "RemoveContainer" containerID="bcd65b23813851dffffe16db781dc6ecf5012a5e1e917f91abbe0e7c508121f8"
Oct 30 22:42:31.259699 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:42:31.259680    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcd65b23813851dffffe16db781dc6ecf5012a5e1e917f91abbe0e7c508121f8\": container with ID starting with bcd65b23813851dffffe16db781dc6ecf5012a5e1e917f91abbe0e7c508121f8 not found: ID does not exist" containerID="bcd65b23813851dffffe16db781dc6ecf5012a5e1e917f91abbe0e7c508121f8"
Oct 30 22:42:31.259746 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.259708    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcd65b23813851dffffe16db781dc6ecf5012a5e1e917f91abbe0e7c508121f8"} err="failed to get container status \"bcd65b23813851dffffe16db781dc6ecf5012a5e1e917f91abbe0e7c508121f8\": rpc error: code = NotFound desc = could not find container \"bcd65b23813851dffffe16db781dc6ecf5012a5e1e917f91abbe0e7c508121f8\": container with ID starting with bcd65b23813851dffffe16db781dc6ecf5012a5e1e917f91abbe0e7c508121f8 not found: ID does not exist"
Oct 30 22:42:31.259746 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.259727    4889 scope.go:122] "RemoveContainer" containerID="d385f1fd17d203664f64a9a612f0007617ba893cef05ea957423d7e0b3cc2f95"
Oct 30 22:42:31.294511 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.294477    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"]
Oct 30 22:42:31.299488 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.299457    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cmklv"]
Oct 30 22:42:31.345334 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.345296    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"]
Oct 30 22:42:31.350385 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.350359    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p65ln"]
Oct 30 22:42:31.847478 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.847442    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"]
Oct 30 22:42:31.848397 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.848376    4889 cpu_manager.go:414] "containerMap: removing container" podUID="246eac1b-e89f-4433-930d-f8adbd60524f" containerName="istio-proxy"
Oct 30 22:42:31.848397 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.848395    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="246eac1b-e89f-4433-930d-f8adbd60524f" containerName="istio-proxy"
Oct 30 22:42:31.848633 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.848452    4889 cpu_manager.go:414] "containerMap: removing container" podUID="31604bd8-b5a2-435f-b021-f4f29b31b8ce" containerName="istio-proxy"
Oct 30 22:42:31.848633 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.848459    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31604bd8-b5a2-435f-b021-f4f29b31b8ce" containerName="istio-proxy"
Oct 30 22:42:31.848633 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.848594    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="31604bd8-b5a2-435f-b021-f4f29b31b8ce" containerName="istio-proxy"
Oct 30 22:42:31.848633 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.848608    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="246eac1b-e89f-4433-930d-f8adbd60524f" containerName="istio-proxy"
Oct 30 22:42:31.855379 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.855350    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:31.875961 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.875929    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"]
Oct 30 22:42:31.880771 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.880739    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3acfdb-openshift-default-dockercfg-bthnn\""
Oct 30 22:42:31.899536 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.899495    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-data\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:31.899694 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.899626    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dcxz\" (UniqueName: \"kubernetes.io/projected/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-kube-api-access-2dcxz\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:31.899761 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.899694    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-workload-certs\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:31.899832 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.899756    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-workload-socket\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:31.899890 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.899857    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-token\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:31.899970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.899948    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-envoy\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:31.900049 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.900032    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-podinfo\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:31.900296 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.900263    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-credential-socket\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:31.900376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.900362    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istiod-ca-cert\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:31.952392 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:31.952299    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:42:32.006326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.002906    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-istio-data\") pod \"7221740b-e003-4af1-86a2-396e32fff7ca\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") "
Oct 30 22:42:32.006326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.003012    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-workload-socket\") pod \"7221740b-e003-4af1-86a2-396e32fff7ca\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") "
Oct 30 22:42:32.006326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.003070    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7221740b-e003-4af1-86a2-396e32fff7ca-kube-api-access-dgbhh\" (UniqueName: \"kubernetes.io/projected/7221740b-e003-4af1-86a2-396e32fff7ca-kube-api-access-dgbhh\") pod \"7221740b-e003-4af1-86a2-396e32fff7ca\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") "
Oct 30 22:42:32.006326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.003104    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7221740b-e003-4af1-86a2-396e32fff7ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7221740b-e003-4af1-86a2-396e32fff7ca-istiod-ca-cert\") pod \"7221740b-e003-4af1-86a2-396e32fff7ca\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") "
Oct 30 22:42:32.006326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.003167    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-credential-socket\") pod \"7221740b-e003-4af1-86a2-396e32fff7ca\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") "
Oct 30 22:42:32.006326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.003303    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-workload-certs\") pod \"7221740b-e003-4af1-86a2-396e32fff7ca\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") "
Oct 30 22:42:32.006326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.003388    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-istio-envoy\") pod \"7221740b-e003-4af1-86a2-396e32fff7ca\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") "
Oct 30 22:42:32.006326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.003427    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7221740b-e003-4af1-86a2-396e32fff7ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7221740b-e003-4af1-86a2-396e32fff7ca-istio-podinfo\") pod \"7221740b-e003-4af1-86a2-396e32fff7ca\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") "
Oct 30 22:42:32.006326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.003463    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7221740b-e003-4af1-86a2-396e32fff7ca-istio-token\" (UniqueName: \"kubernetes.io/projected/7221740b-e003-4af1-86a2-396e32fff7ca-istio-token\") pod \"7221740b-e003-4af1-86a2-396e32fff7ca\" (UID: \"7221740b-e003-4af1-86a2-396e32fff7ca\") "
Oct 30 22:42:32.006326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.004087    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7221740b-e003-4af1-86a2-396e32fff7ca-istiod-ca-cert" pod "7221740b-e003-4af1-86a2-396e32fff7ca" (UID: "7221740b-e003-4af1-86a2-396e32fff7ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:32.006326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.004342    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-istio-data" pod "7221740b-e003-4af1-86a2-396e32fff7ca" (UID: "7221740b-e003-4af1-86a2-396e32fff7ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:32.006326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.004504    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-workload-socket" pod "7221740b-e003-4af1-86a2-396e32fff7ca" (UID: "7221740b-e003-4af1-86a2-396e32fff7ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:32.006326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.005000    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-token\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:32.006326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.005111    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-envoy\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:32.006326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.005176    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-podinfo\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:32.006326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.005243    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-credential-socket\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:32.006326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.005290    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istiod-ca-cert\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:32.006326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.005458    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-data\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:32.006326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.005516    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dcxz\" (UniqueName: \"kubernetes.io/projected/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-kube-api-access-2dcxz\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:32.006326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.005560    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-workload-certs\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:32.006326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.005613    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-workload-socket\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:32.006326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.005732    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7221740b-e003-4af1-86a2-396e32fff7ca-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:32.006326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.005754    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:32.006326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.005768    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:32.006326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.006092    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-workload-socket\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:32.006326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.006314    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-credential-socket" pod "7221740b-e003-4af1-86a2-396e32fff7ca" (UID: "7221740b-e003-4af1-86a2-396e32fff7ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:32.031959 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.006504    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-workload-certs" pod "7221740b-e003-4af1-86a2-396e32fff7ca" (UID: "7221740b-e003-4af1-86a2-396e32fff7ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:32.031959 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.018418    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-istio-envoy" pod "7221740b-e003-4af1-86a2-396e32fff7ca" (UID: "7221740b-e003-4af1-86a2-396e32fff7ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:32.031959 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.022728    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-credential-socket\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:32.050282 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.049385    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7221740b-e003-4af1-86a2-396e32fff7ca-kube-api-access-dgbhh" pod "7221740b-e003-4af1-86a2-396e32fff7ca" (UID: "7221740b-e003-4af1-86a2-396e32fff7ca"). InnerVolumeSpecName "kube-api-access-dgbhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:32.050651 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.050625    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-data\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:32.051063 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.051034    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-workload-certs\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:32.052322 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.052292    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istiod-ca-cert\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:32.053043 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.053018    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7221740b-e003-4af1-86a2-396e32fff7ca-istio-podinfo" pod "7221740b-e003-4af1-86a2-396e32fff7ca" (UID: "7221740b-e003-4af1-86a2-396e32fff7ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:42:32.053790 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.053756    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-envoy\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:32.060401 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.060042    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-podinfo\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:32.060749 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.060716    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7221740b-e003-4af1-86a2-396e32fff7ca-istio-token" pod "7221740b-e003-4af1-86a2-396e32fff7ca" (UID: "7221740b-e003-4af1-86a2-396e32fff7ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:32.067918 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.067888    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:42:32.072665 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.072629    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-token\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:32.074461 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.074435    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dcxz\" (UniqueName: \"kubernetes.io/projected/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-kube-api-access-2dcxz\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:32.101638 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.101602    4889 patch_prober.go:29] interesting pod/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 22:42:32.101805 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.101662    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv" podUID="6e842f91-5282-4f41-b149-d8ce4c8625a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 22:42:32.154369 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.153431    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-workload-socket\") pod \"137329f9-9437-46ab-b030-ead038ccd542\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") "
Oct 30 22:42:32.154369 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.153505    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/137329f9-9437-46ab-b030-ead038ccd542-kube-api-access-f8q27\" (UniqueName: \"kubernetes.io/projected/137329f9-9437-46ab-b030-ead038ccd542-kube-api-access-f8q27\") pod \"137329f9-9437-46ab-b030-ead038ccd542\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") "
Oct 30 22:42:32.154369 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.153667    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/137329f9-9437-46ab-b030-ead038ccd542-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137329f9-9437-46ab-b030-ead038ccd542-istio-podinfo\") pod \"137329f9-9437-46ab-b030-ead038ccd542\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") "
Oct 30 22:42:32.154369 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.153849    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/137329f9-9437-46ab-b030-ead038ccd542-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137329f9-9437-46ab-b030-ead038ccd542-istiod-ca-cert\") pod \"137329f9-9437-46ab-b030-ead038ccd542\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") "
Oct 30 22:42:32.154369 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.154114    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-workload-certs\") pod \"137329f9-9437-46ab-b030-ead038ccd542\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") "
Oct 30 22:42:32.154788 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.154555    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-istio-data\") pod \"137329f9-9437-46ab-b030-ead038ccd542\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") "
Oct 30 22:42:32.154929 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.154867    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/137329f9-9437-46ab-b030-ead038ccd542-istio-token\" (UniqueName: \"kubernetes.io/projected/137329f9-9437-46ab-b030-ead038ccd542-istio-token\") pod \"137329f9-9437-46ab-b030-ead038ccd542\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") "
Oct 30 22:42:32.155170 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.155148    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-istio-envoy\") pod \"137329f9-9437-46ab-b030-ead038ccd542\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") "
Oct 30 22:42:32.155469 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.155445    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-credential-socket\") pod \"137329f9-9437-46ab-b030-ead038ccd542\" (UID: \"137329f9-9437-46ab-b030-ead038ccd542\") "
Oct 30 22:42:32.159405 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.159213    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137329f9-9437-46ab-b030-ead038ccd542-istiod-ca-cert" pod "137329f9-9437-46ab-b030-ead038ccd542" (UID: "137329f9-9437-46ab-b030-ead038ccd542"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:32.159643 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.159620    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-workload-socket" pod "137329f9-9437-46ab-b030-ead038ccd542" (UID: "137329f9-9437-46ab-b030-ead038ccd542"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:32.164189 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.164150    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-istio-data" pod "137329f9-9437-46ab-b030-ead038ccd542" (UID: "137329f9-9437-46ab-b030-ead038ccd542"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:32.165353 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.164414    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-workload-certs" pod "137329f9-9437-46ab-b030-ead038ccd542" (UID: "137329f9-9437-46ab-b030-ead038ccd542"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:32.168967 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.167331    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:32.168967 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.167363    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137329f9-9437-46ab-b030-ead038ccd542-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:32.168967 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.167385    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:32.168967 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.167398    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7221740b-e003-4af1-86a2-396e32fff7ca-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:32.168967 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.167566    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7221740b-e003-4af1-86a2-396e32fff7ca-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:32.168967 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.167580    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgbhh\" (UniqueName: \"kubernetes.io/projected/7221740b-e003-4af1-86a2-396e32fff7ca-kube-api-access-dgbhh\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:32.168967 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.167602    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7221740b-e003-4af1-86a2-396e32fff7ca-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:32.168967 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.167739    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:32.181969 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.181901    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-credential-socket" pod "137329f9-9437-46ab-b030-ead038ccd542" (UID: "137329f9-9437-46ab-b030-ead038ccd542"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:32.187275 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.185788    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137329f9-9437-46ab-b030-ead038ccd542-istio-token" pod "137329f9-9437-46ab-b030-ead038ccd542" (UID: "137329f9-9437-46ab-b030-ead038ccd542"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:32.187275 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.186012    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-istio-envoy" pod "137329f9-9437-46ab-b030-ead038ccd542" (UID: "137329f9-9437-46ab-b030-ead038ccd542"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:32.187275 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.186023    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/137329f9-9437-46ab-b030-ead038ccd542-istio-podinfo" pod "137329f9-9437-46ab-b030-ead038ccd542" (UID: "137329f9-9437-46ab-b030-ead038ccd542"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:42:32.187275 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.186397    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137329f9-9437-46ab-b030-ead038ccd542-kube-api-access-f8q27" pod "137329f9-9437-46ab-b030-ead038ccd542" (UID: "137329f9-9437-46ab-b030-ead038ccd542"). InnerVolumeSpecName "kube-api-access-f8q27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:32.192333 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.192300    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:32.260812 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.260777    4889 generic.go:356] "Generic (PLEG): container finished" podID="137329f9-9437-46ab-b030-ead038ccd542" containerID="c49ea9e2f2ae9fc1a105e6a0647ad8743c92428eea6417eeb109bb0a6837ba3c" exitCode=0
Oct 30 22:42:32.260991 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.260895    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt" event={"ID":"137329f9-9437-46ab-b030-ead038ccd542","Type":"ContainerDied","Data":"c49ea9e2f2ae9fc1a105e6a0647ad8743c92428eea6417eeb109bb0a6837ba3c"}
Oct 30 22:42:32.260991 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.260975    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt" event={"ID":"137329f9-9437-46ab-b030-ead038ccd542","Type":"ContainerDied","Data":"a1f7b76fc3bbed2d41d338873eb5d37bf25e8b6c1a8028c05601652c07aa7a10"}
Oct 30 22:42:32.261152 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.261007    4889 scope.go:122] "RemoveContainer" containerID="c49ea9e2f2ae9fc1a105e6a0647ad8743c92428eea6417eeb109bb0a6837ba3c"
Oct 30 22:42:32.261277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.261253    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"
Oct 30 22:42:32.270821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.270790    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8q27\" (UniqueName: \"kubernetes.io/projected/137329f9-9437-46ab-b030-ead038ccd542-kube-api-access-f8q27\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:32.270821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.270823    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137329f9-9437-46ab-b030-ead038ccd542-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:32.271075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.270838    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:32.271075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.270850    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:32.271075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.270863    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137329f9-9437-46ab-b030-ead038ccd542-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:32.271075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.270876    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:32.271075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.270886    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137329f9-9437-46ab-b030-ead038ccd542-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:32.274897 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.274864    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg" event={"ID":"7221740b-e003-4af1-86a2-396e32fff7ca","Type":"ContainerDied","Data":"ea4bbe3ee57db79ddd8e0b2bd81f388b4acac2ee5c6ed42846a3e2bdb01b651c"}
Oct 30 22:42:32.275034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.274908    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"
Oct 30 22:42:32.286308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.284073    4889 scope.go:122] "RemoveContainer" containerID="b95598dfcf28df436f7a2d55ed03f23337e74d6cf084ac26b9ceca13f16a5dc0"
Oct 30 22:42:32.299600 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.299576    4889 scope.go:122] "RemoveContainer" containerID="c49ea9e2f2ae9fc1a105e6a0647ad8743c92428eea6417eeb109bb0a6837ba3c"
Oct 30 22:42:32.300004 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:42:32.299974    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c49ea9e2f2ae9fc1a105e6a0647ad8743c92428eea6417eeb109bb0a6837ba3c\": container with ID starting with c49ea9e2f2ae9fc1a105e6a0647ad8743c92428eea6417eeb109bb0a6837ba3c not found: ID does not exist" containerID="c49ea9e2f2ae9fc1a105e6a0647ad8743c92428eea6417eeb109bb0a6837ba3c"
Oct 30 22:42:32.300078 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.300030    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49ea9e2f2ae9fc1a105e6a0647ad8743c92428eea6417eeb109bb0a6837ba3c"} err="failed to get container status \"c49ea9e2f2ae9fc1a105e6a0647ad8743c92428eea6417eeb109bb0a6837ba3c\": rpc error: code = NotFound desc = could not find container \"c49ea9e2f2ae9fc1a105e6a0647ad8743c92428eea6417eeb109bb0a6837ba3c\": container with ID starting with c49ea9e2f2ae9fc1a105e6a0647ad8743c92428eea6417eeb109bb0a6837ba3c not found: ID does not exist"
Oct 30 22:42:32.300078 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.300055    4889 scope.go:122] "RemoveContainer" containerID="b95598dfcf28df436f7a2d55ed03f23337e74d6cf084ac26b9ceca13f16a5dc0"
Oct 30 22:42:32.300791 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:42:32.300765    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b95598dfcf28df436f7a2d55ed03f23337e74d6cf084ac26b9ceca13f16a5dc0\": container with ID starting with b95598dfcf28df436f7a2d55ed03f23337e74d6cf084ac26b9ceca13f16a5dc0 not found: ID does not exist" containerID="b95598dfcf28df436f7a2d55ed03f23337e74d6cf084ac26b9ceca13f16a5dc0"
Oct 30 22:42:32.300935 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.300798    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95598dfcf28df436f7a2d55ed03f23337e74d6cf084ac26b9ceca13f16a5dc0"} err="failed to get container status \"b95598dfcf28df436f7a2d55ed03f23337e74d6cf084ac26b9ceca13f16a5dc0\": rpc error: code = NotFound desc = could not find container \"b95598dfcf28df436f7a2d55ed03f23337e74d6cf084ac26b9ceca13f16a5dc0\": container with ID starting with b95598dfcf28df436f7a2d55ed03f23337e74d6cf084ac26b9ceca13f16a5dc0 not found: ID does not exist"
Oct 30 22:42:32.300935 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.300818    4889 scope.go:122] "RemoveContainer" containerID="b9bef27d10252e835d86a70b9b33765d31305448b82624b38a8434991f107d55"
Oct 30 22:42:32.406855 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.406756    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"]
Oct 30 22:42:32.437564 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.437523    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-dxslt"]
Oct 30 22:42:32.465426 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.465381    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"]
Oct 30 22:42:32.470243 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.470196    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-8dgzg"]
Oct 30 22:42:32.593682 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.593524    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"]
Oct 30 22:42:32.597337 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:42:32.597301    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ab0e7b_53c9_4cc3_ba1c_8e3f8f3ac5a8.slice/crio-08d86dd84f7152bb337be1f701771c320828066ae704cd6667d7c20f47001832 WatchSource:0}: Error finding container 08d86dd84f7152bb337be1f701771c320828066ae704cd6667d7c20f47001832: Status 404 returned error can't find the container with id 08d86dd84f7152bb337be1f701771c320828066ae704cd6667d7c20f47001832
Oct 30 22:42:32.601311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.601268    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:42:32.601415 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.601369    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:42:32.601463 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.601415    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:42:32.778140 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.778108    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="137329f9-9437-46ab-b030-ead038ccd542" path="/var/lib/kubelet/pods/137329f9-9437-46ab-b030-ead038ccd542/volumes"
Oct 30 22:42:32.778824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.778798    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="246eac1b-e89f-4433-930d-f8adbd60524f" path="/var/lib/kubelet/pods/246eac1b-e89f-4433-930d-f8adbd60524f/volumes"
Oct 30 22:42:32.779438 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.779416    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31604bd8-b5a2-435f-b021-f4f29b31b8ce" path="/var/lib/kubelet/pods/31604bd8-b5a2-435f-b021-f4f29b31b8ce/volumes"
Oct 30 22:42:32.780295 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.780274    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7221740b-e003-4af1-86a2-396e32fff7ca" path="/var/lib/kubelet/pods/7221740b-e003-4af1-86a2-396e32fff7ca/volumes"
Oct 30 22:42:32.943742 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:32.936367    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:33.101557 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:33.101522    4889 patch_prober.go:29] interesting pod/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 22:42:33.102032 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:33.101601    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv" podUID="6e842f91-5282-4f41-b149-d8ce4c8625a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 22:42:33.287835 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:33.287793    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s" event={"ID":"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8","Type":"ContainerStarted","Data":"e8c2089231571ba84c3f9802a2814d7d3c1902509eea0e82f6b9d3e07722e922"}
Oct 30 22:42:33.288056 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:33.287868    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s" event={"ID":"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8","Type":"ContainerStarted","Data":"08d86dd84f7152bb337be1f701771c320828066ae704cd6667d7c20f47001832"}
Oct 30 22:42:33.345358 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:33.345299    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s" podStartSLOduration=2.345279981 podStartE2EDuration="2.345279981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:33.335735439 +0000 UTC m=+2138.692101756" watchObservedRunningTime="2026-10-30 22:42:33.345279981 +0000 UTC m=+2138.701646295"
Oct 30 22:42:34.102175 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:34.102135    4889 patch_prober.go:29] interesting pod/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 22:42:34.102638 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:34.102197    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv" podUID="6e842f91-5282-4f41-b149-d8ce4c8625a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 22:42:34.193585 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:34.193545    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:34.195405 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:34.195374    4889 patch_prober.go:29] interesting pod/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 22:42:34.195563 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:34.195425    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s" podUID="23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 22:42:34.951250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:34.948877    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:35.108149 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:35.108117    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:35.108734 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:35.108712    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:35.125812 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:35.125774    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:42:35.193627 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:35.193585    4889 patch_prober.go:29] interesting pod/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 22:42:35.193841 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:35.193646    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s" podUID="23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 22:42:35.938457 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:35.938416    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:36.127252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:36.109456    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:36.192069 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:36.191972    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"]
Oct 30 22:42:36.192300 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:36.192273    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795" podUID="007bdaf6-3319-49db-850b-7fd9b87a5b7a" containerName="istio-proxy" containerID="cri-o://4d6e6d920484364b8ef55f8cf0213339ad6e8348219075bc3d2300855cb95f85" gracePeriod=30
Oct 30 22:42:36.193075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:36.193044    4889 patch_prober.go:29] interesting pod/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 22:42:36.193293 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:36.193267    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s" podUID="23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 22:42:36.203759 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:36.203714    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"]
Oct 30 22:42:36.204022 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:36.203981    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z" podUID="8b34996e-28a0-4cbf-b0fa-4615ad9fccaa" containerName="istio-proxy" containerID="cri-o://02b2df1c5dfc1189bea3c629356c1ceae2b6ae0064ac550a33166e01262c0764" gracePeriod=30
Oct 30 22:42:36.243694 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:36.243546    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"]
Oct 30 22:42:36.244274 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:36.244141    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6" podUID="b211a3e6-ec7c-418c-9fa1-a2546cacd7f7" containerName="istio-proxy" containerID="cri-o://1f1b7d34f491636176e9f6fcd1d55b356b2b4a8da3a55c76055e35d1daa198a8" gracePeriod=30
Oct 30 22:42:36.295643 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:36.295540    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:37.192141 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:37.192100    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:37.198051 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:37.198015    4889 patch_prober.go:29] interesting pod/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 22:42:37.201553 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:37.201505    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s" podUID="23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 22:42:38.035594 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:38.035554    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq" podUID="c24e6bd5-4b2f-4060-b4cb-b2df03309b3a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:42:38.146217 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:38.146179    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:42:38.146403 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:38.146255    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6" podUID="b211a3e6-ec7c-418c-9fa1-a2546cacd7f7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:42:38.193791 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:38.193754    4889 patch_prober.go:29] interesting pod/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 22:42:38.194185 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:38.193819    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s" podUID="23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 22:42:38.498446 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:38.498412    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:42:38.498612 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:38.498478    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795" podUID="007bdaf6-3319-49db-850b-7fd9b87a5b7a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:42:39.193556 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:39.193520    4889 patch_prober.go:29] interesting pod/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 22:42:39.193822 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:39.193581    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s" podUID="23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 22:42:39.255544 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:39.255501    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:40.198864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:40.198833    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:40.199366 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:40.199331    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:40.200039 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:40.200016    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:42:40.304487 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:42:40.304439    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:42:41.060117 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.060078    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:41.451879 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.451844    4889 generic.go:356] "Generic (PLEG): container finished" podID="b211a3e6-ec7c-418c-9fa1-a2546cacd7f7" containerID="1f1b7d34f491636176e9f6fcd1d55b356b2b4a8da3a55c76055e35d1daa198a8" exitCode=0
Oct 30 22:42:41.457623 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.452428    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6" event={"ID":"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7","Type":"ContainerDied","Data":"1f1b7d34f491636176e9f6fcd1d55b356b2b4a8da3a55c76055e35d1daa198a8"}
Oct 30 22:42:41.460854 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.460826    4889 generic.go:356] "Generic (PLEG): container finished" podID="8b34996e-28a0-4cbf-b0fa-4615ad9fccaa" containerID="02b2df1c5dfc1189bea3c629356c1ceae2b6ae0064ac550a33166e01262c0764" exitCode=0
Oct 30 22:42:41.461012 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.460861    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z" event={"ID":"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa","Type":"ContainerDied","Data":"02b2df1c5dfc1189bea3c629356c1ceae2b6ae0064ac550a33166e01262c0764"}
Oct 30 22:42:41.462104 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.461338    4889 scope.go:122] "RemoveContainer" containerID="13a8216f5efc056073c38bf635471fcaae2938ab56d8d1df481b9d5dc2188a75"
Oct 30 22:42:41.466465 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.465175    4889 generic.go:356] "Generic (PLEG): container finished" podID="007bdaf6-3319-49db-850b-7fd9b87a5b7a" containerID="4d6e6d920484364b8ef55f8cf0213339ad6e8348219075bc3d2300855cb95f85" exitCode=0
Oct 30 22:42:41.466465 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.465935    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795" event={"ID":"007bdaf6-3319-49db-850b-7fd9b87a5b7a","Type":"ContainerDied","Data":"4d6e6d920484364b8ef55f8cf0213339ad6e8348219075bc3d2300855cb95f85"}
Oct 30 22:42:41.564993 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.564961    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:42:41.610761 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.610730    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:42:41.643509 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.643481    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:42:41.739498 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.739466    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-podinfo\") pod \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") "
Oct 30 22:42:41.739677 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.739533    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-workload-certs\") pod \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") "
Oct 30 22:42:41.739677 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.739555    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istiod-ca-cert\") pod \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") "
Oct 30 22:42:41.739677 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.739602    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-podinfo\") pod \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") "
Oct 30 22:42:41.739677 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.739619    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-credential-socket\") pod \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") "
Oct 30 22:42:41.739905 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.739683    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-workload-socket\") pod \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") "
Oct 30 22:42:41.739905 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.739728    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-kube-api-access-vwv79\" (UniqueName: \"kubernetes.io/projected/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-kube-api-access-vwv79\") pod \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") "
Oct 30 22:42:41.739905 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.739746    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-envoy\") pod \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") "
Oct 30 22:42:41.739905 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.739781    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istiod-ca-cert\") pod \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") "
Oct 30 22:42:41.739905 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.739799    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-envoy\") pod \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") "
Oct 30 22:42:41.739905 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.739820    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-token\" (UniqueName: \"kubernetes.io/projected/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-token\") pod \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") "
Oct 30 22:42:41.739905 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.739841    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-data\") pod \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") "
Oct 30 22:42:41.740376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.739936    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-workload-certs\") pod \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") "
Oct 30 22:42:41.740376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.739960    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-workload-socket\") pod \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") "
Oct 30 22:42:41.740376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.739986    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-data\") pod \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") "
Oct 30 22:42:41.740376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.740014    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-credential-socket\") pod \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") "
Oct 30 22:42:41.740376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.740039    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-token\" (UniqueName: \"kubernetes.io/projected/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-token\") pod \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\" (UID: \"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa\") "
Oct 30 22:42:41.740376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.740087    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/007bdaf6-3319-49db-850b-7fd9b87a5b7a-kube-api-access-vmlxk\" (UniqueName: \"kubernetes.io/projected/007bdaf6-3319-49db-850b-7fd9b87a5b7a-kube-api-access-vmlxk\") pod \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\" (UID: \"007bdaf6-3319-49db-850b-7fd9b87a5b7a\") "
Oct 30 22:42:41.740683 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.740592    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-workload-socket" pod "007bdaf6-3319-49db-850b-7fd9b87a5b7a" (UID: "007bdaf6-3319-49db-850b-7fd9b87a5b7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.740739 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.740703    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istiod-ca-cert" pod "8b34996e-28a0-4cbf-b0fa-4615ad9fccaa" (UID: "8b34996e-28a0-4cbf-b0fa-4615ad9fccaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:41.740952 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.740922    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-credential-socket" pod "007bdaf6-3319-49db-850b-7fd9b87a5b7a" (UID: "007bdaf6-3319-49db-850b-7fd9b87a5b7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.740952 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.740933    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-workload-certs" pod "007bdaf6-3319-49db-850b-7fd9b87a5b7a" (UID: "007bdaf6-3319-49db-850b-7fd9b87a5b7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.741454 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.741429    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-workload-certs" pod "8b34996e-28a0-4cbf-b0fa-4615ad9fccaa" (UID: "8b34996e-28a0-4cbf-b0fa-4615ad9fccaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.741454 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.741447    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.741620 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.741460    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-data" pod "8b34996e-28a0-4cbf-b0fa-4615ad9fccaa" (UID: "8b34996e-28a0-4cbf-b0fa-4615ad9fccaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.741620 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.741470    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.741620 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.741497    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.741620 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.741515    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.741852 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.741670    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-credential-socket" pod "8b34996e-28a0-4cbf-b0fa-4615ad9fccaa" (UID: "8b34996e-28a0-4cbf-b0fa-4615ad9fccaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.741852 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.741714    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-workload-socket" pod "8b34996e-28a0-4cbf-b0fa-4615ad9fccaa" (UID: "8b34996e-28a0-4cbf-b0fa-4615ad9fccaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.742046 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.742025    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istiod-ca-cert" pod "007bdaf6-3319-49db-850b-7fd9b87a5b7a" (UID: "007bdaf6-3319-49db-850b-7fd9b87a5b7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:41.743652 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.743626    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-data" pod "007bdaf6-3319-49db-850b-7fd9b87a5b7a" (UID: "007bdaf6-3319-49db-850b-7fd9b87a5b7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.744706 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.744601    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-podinfo" pod "8b34996e-28a0-4cbf-b0fa-4615ad9fccaa" (UID: "8b34996e-28a0-4cbf-b0fa-4615ad9fccaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:42:41.744845 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.744752    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007bdaf6-3319-49db-850b-7fd9b87a5b7a-kube-api-access-vmlxk" pod "007bdaf6-3319-49db-850b-7fd9b87a5b7a" (UID: "007bdaf6-3319-49db-850b-7fd9b87a5b7a"). InnerVolumeSpecName "kube-api-access-vmlxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:41.744959 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.744931    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-envoy" pod "8b34996e-28a0-4cbf-b0fa-4615ad9fccaa" (UID: "8b34996e-28a0-4cbf-b0fa-4615ad9fccaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.745377 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.745353    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-podinfo" pod "007bdaf6-3319-49db-850b-7fd9b87a5b7a" (UID: "007bdaf6-3319-49db-850b-7fd9b87a5b7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:42:41.745706 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.745682    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-envoy" pod "007bdaf6-3319-49db-850b-7fd9b87a5b7a" (UID: "007bdaf6-3319-49db-850b-7fd9b87a5b7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.745771 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.745706    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-kube-api-access-vwv79" pod "8b34996e-28a0-4cbf-b0fa-4615ad9fccaa" (UID: "8b34996e-28a0-4cbf-b0fa-4615ad9fccaa"). InnerVolumeSpecName "kube-api-access-vwv79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:41.746134 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.746107    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-token" pod "007bdaf6-3319-49db-850b-7fd9b87a5b7a" (UID: "007bdaf6-3319-49db-850b-7fd9b87a5b7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:41.746604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.746581    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-token" pod "8b34996e-28a0-4cbf-b0fa-4615ad9fccaa" (UID: "8b34996e-28a0-4cbf-b0fa-4615ad9fccaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:41.843023 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.842927    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-credential-socket\") pod \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") "
Oct 30 22:42:41.843023 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.842975    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-data\") pod \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") "
Oct 30 22:42:41.843023 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.843002    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-kube-api-access-4z6zl\" (UniqueName: \"kubernetes.io/projected/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-kube-api-access-4z6zl\") pod \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") "
Oct 30 22:42:41.843400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.843030    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istiod-ca-cert\") pod \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") "
Oct 30 22:42:41.843400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.843091    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-envoy\") pod \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") "
Oct 30 22:42:41.843400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.843117    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-token\" (UniqueName: \"kubernetes.io/projected/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-token\") pod \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") "
Oct 30 22:42:41.843400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.843153    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-workload-socket\") pod \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") "
Oct 30 22:42:41.843400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.843239    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-credential-socket" pod "b211a3e6-ec7c-418c-9fa1-a2546cacd7f7" (UID: "b211a3e6-ec7c-418c-9fa1-a2546cacd7f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.843400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.843282    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-data" pod "b211a3e6-ec7c-418c-9fa1-a2546cacd7f7" (UID: "b211a3e6-ec7c-418c-9fa1-a2546cacd7f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.843400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.843374    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-podinfo\") pod \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") "
Oct 30 22:42:41.843873 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.843483    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-workload-certs\") pod \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\" (UID: \"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7\") "
Oct 30 22:42:41.843873 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.843618    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-workload-socket" pod "b211a3e6-ec7c-418c-9fa1-a2546cacd7f7" (UID: "b211a3e6-ec7c-418c-9fa1-a2546cacd7f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.843873 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.843758    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istiod-ca-cert" pod "b211a3e6-ec7c-418c-9fa1-a2546cacd7f7" (UID: "b211a3e6-ec7c-418c-9fa1-a2546cacd7f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:41.844098 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.844057    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-workload-certs" pod "b211a3e6-ec7c-418c-9fa1-a2546cacd7f7" (UID: "b211a3e6-ec7c-418c-9fa1-a2546cacd7f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.844667 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.844634    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmlxk\" (UniqueName: \"kubernetes.io/projected/007bdaf6-3319-49db-850b-7fd9b87a5b7a-kube-api-access-vmlxk\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.844667 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.844666    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.844840 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.844682    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.844840 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.844696    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.844840 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.844709    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.844840 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.844723    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.844840 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.844735    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.844840 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.844748    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.844840 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.844760    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.844840 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.844775    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwv79\" (UniqueName: \"kubernetes.io/projected/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-kube-api-access-vwv79\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.844840 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.844787    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.844840 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.844801    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.844840 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.844813    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.844840 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.844825    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007bdaf6-3319-49db-850b-7fd9b87a5b7a-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.844840 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.844837    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.845400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.844851    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.845400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.844864    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.845400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.844876    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.845400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.844892    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.846235 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.846198    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-kube-api-access-4z6zl" pod "b211a3e6-ec7c-418c-9fa1-a2546cacd7f7" (UID: "b211a3e6-ec7c-418c-9fa1-a2546cacd7f7"). InnerVolumeSpecName "kube-api-access-4z6zl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:41.847053 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.847027    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-envoy" pod "b211a3e6-ec7c-418c-9fa1-a2546cacd7f7" (UID: "b211a3e6-ec7c-418c-9fa1-a2546cacd7f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.847146 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.847122    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-podinfo" pod "b211a3e6-ec7c-418c-9fa1-a2546cacd7f7" (UID: "b211a3e6-ec7c-418c-9fa1-a2546cacd7f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:42:41.847187 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.847144    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-token" pod "b211a3e6-ec7c-418c-9fa1-a2546cacd7f7" (UID: "b211a3e6-ec7c-418c-9fa1-a2546cacd7f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:41.946644 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.946613    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.946644 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.946639    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z6zl\" (UniqueName: \"kubernetes.io/projected/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-kube-api-access-4z6zl\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.946644 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.946649    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.946870 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:41.946659    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:42.474641 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:42.474553    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"
Oct 30 22:42:42.474641 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:42.474579    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6" event={"ID":"b211a3e6-ec7c-418c-9fa1-a2546cacd7f7","Type":"ContainerDied","Data":"75e589eba7ca7acfae7e2540f468ae578d3e3d607275f3050dd49c83345fd43d"}
Oct 30 22:42:42.474641 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:42.474634    4889 scope.go:122] "RemoveContainer" containerID="1f1b7d34f491636176e9f6fcd1d55b356b2b4a8da3a55c76055e35d1daa198a8"
Oct 30 22:42:42.476496 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:42.476480    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"
Oct 30 22:42:42.476639 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:42.476527    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z" event={"ID":"8b34996e-28a0-4cbf-b0fa-4615ad9fccaa","Type":"ContainerDied","Data":"9944626a087204c0c1cf4cb5638d6fe97029bbe1fa954d7a7432be80c67ff7d0"}
Oct 30 22:42:42.478265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:42.478247    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"
Oct 30 22:42:42.478265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:42.478215    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795" event={"ID":"007bdaf6-3319-49db-850b-7fd9b87a5b7a","Type":"ContainerDied","Data":"bb6838e1d46e5ead12096ae1c6398979f74ec2b4b962c8f7c52b71947ad69e7b"}
Oct 30 22:42:42.490098 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:42.490073    4889 scope.go:122] "RemoveContainer" containerID="02b2df1c5dfc1189bea3c629356c1ceae2b6ae0064ac550a33166e01262c0764"
Oct 30 22:42:42.502211 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:42.502192    4889 scope.go:122] "RemoveContainer" containerID="4d6e6d920484364b8ef55f8cf0213339ad6e8348219075bc3d2300855cb95f85"
Oct 30 22:42:42.550499 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:42.550468    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"]
Oct 30 22:42:42.564005 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:42.563980    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-q5795"]
Oct 30 22:42:42.618065 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:42.618035    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"]
Oct 30 22:42:42.622446 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:42.622423    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-dbxb6"]
Oct 30 22:42:42.659996 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:42.659966    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"]
Oct 30 22:42:42.672932 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:42.672903    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-bf77z"]
Oct 30 22:42:42.777237 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:42.777196    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="007bdaf6-3319-49db-850b-7fd9b87a5b7a" path="/var/lib/kubelet/pods/007bdaf6-3319-49db-850b-7fd9b87a5b7a/volumes"
Oct 30 22:42:42.777913 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:42.777894    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b34996e-28a0-4cbf-b0fa-4615ad9fccaa" path="/var/lib/kubelet/pods/8b34996e-28a0-4cbf-b0fa-4615ad9fccaa/volumes"
Oct 30 22:42:42.778393 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:42.778374    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b211a3e6-ec7c-418c-9fa1-a2546cacd7f7" path="/var/lib/kubelet/pods/b211a3e6-ec7c-418c-9fa1-a2546cacd7f7/volumes"
Oct 30 22:42:42.941319 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:42.941285    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:44.687257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:44.686696    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:46.467249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:46.465755    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:47.424849 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:47.424815    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5"]
Oct 30 22:42:47.630387 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:47.630350    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:47.894950 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:47.894909    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:48.030265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:48.030190    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq" podUID="c24e6bd5-4b2f-4060-b4cb-b2df03309b3a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:42:48.393308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:48.392198    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:49.101469 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:49.101437    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:50.310064 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:50.310021    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:51.374206 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:51.374178    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq"
Oct 30 22:42:51.385261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:51.385207    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c24e6bd5-4b2f-4060-b4cb-b2df03309b3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24e6bd5-4b2f-4060-b4cb-b2df03309b3a-plugins-config-volume\") pod \"c24e6bd5-4b2f-4060-b4cb-b2df03309b3a\" (UID: \"c24e6bd5-4b2f-4060-b4cb-b2df03309b3a\") "
Oct 30 22:42:51.385404 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:51.385367    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c24e6bd5-4b2f-4060-b4cb-b2df03309b3a-kube-api-access-g8p58\" (UniqueName: \"kubernetes.io/projected/c24e6bd5-4b2f-4060-b4cb-b2df03309b3a-kube-api-access-g8p58\") pod \"c24e6bd5-4b2f-4060-b4cb-b2df03309b3a\" (UID: \"c24e6bd5-4b2f-4060-b4cb-b2df03309b3a\") "
Oct 30 22:42:51.385545 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:51.385524    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c24e6bd5-4b2f-4060-b4cb-b2df03309b3a-plugins-config-volume" pod "c24e6bd5-4b2f-4060-b4cb-b2df03309b3a" (UID: "c24e6bd5-4b2f-4060-b4cb-b2df03309b3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:51.386130 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:51.386104    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24e6bd5-4b2f-4060-b4cb-b2df03309b3a-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:51.387548 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:51.387526    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24e6bd5-4b2f-4060-b4cb-b2df03309b3a-kube-api-access-g8p58" pod "c24e6bd5-4b2f-4060-b4cb-b2df03309b3a" (UID: "c24e6bd5-4b2f-4060-b4cb-b2df03309b3a"). InnerVolumeSpecName "kube-api-access-g8p58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:51.487553 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:51.487519    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8p58\" (UniqueName: \"kubernetes.io/projected/c24e6bd5-4b2f-4060-b4cb-b2df03309b3a-kube-api-access-g8p58\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:42:51.565290 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:51.565252    4889 generic.go:356] "Generic (PLEG): container finished" podID="c24e6bd5-4b2f-4060-b4cb-b2df03309b3a" containerID="75dccf023d12f60a21d74c5e905a437b983724c19129e41b7229fd55cf5b1e48" exitCode=0
Oct 30 22:42:51.565290 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:51.565282    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq"
Oct 30 22:42:51.565628 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:51.565324    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq" event={"ID":"c24e6bd5-4b2f-4060-b4cb-b2df03309b3a","Type":"ContainerDied","Data":"75dccf023d12f60a21d74c5e905a437b983724c19129e41b7229fd55cf5b1e48"}
Oct 30 22:42:51.565628 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:51.565373    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq" event={"ID":"c24e6bd5-4b2f-4060-b4cb-b2df03309b3a","Type":"ContainerDied","Data":"e1ac1d1bce1c785dc466b6e9ad9cf614daa1bf862146b95a6e4cbcfbae81c0f8"}
Oct 30 22:42:51.565628 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:51.565397    4889 scope.go:122] "RemoveContainer" containerID="75dccf023d12f60a21d74c5e905a437b983724c19129e41b7229fd55cf5b1e48"
Oct 30 22:42:51.577156 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:51.577139    4889 scope.go:122] "RemoveContainer" containerID="75dccf023d12f60a21d74c5e905a437b983724c19129e41b7229fd55cf5b1e48"
Oct 30 22:42:51.577437 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:42:51.577419    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75dccf023d12f60a21d74c5e905a437b983724c19129e41b7229fd55cf5b1e48\": container with ID starting with 75dccf023d12f60a21d74c5e905a437b983724c19129e41b7229fd55cf5b1e48 not found: ID does not exist" containerID="75dccf023d12f60a21d74c5e905a437b983724c19129e41b7229fd55cf5b1e48"
Oct 30 22:42:51.577499 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:51.577445    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75dccf023d12f60a21d74c5e905a437b983724c19129e41b7229fd55cf5b1e48"} err="failed to get container status \"75dccf023d12f60a21d74c5e905a437b983724c19129e41b7229fd55cf5b1e48\": rpc error: code = NotFound desc = could not find container \"75dccf023d12f60a21d74c5e905a437b983724c19129e41b7229fd55cf5b1e48\": container with ID starting with 75dccf023d12f60a21d74c5e905a437b983724c19129e41b7229fd55cf5b1e48 not found: ID does not exist"
Oct 30 22:42:51.631354 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:51.631263    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq"]
Oct 30 22:42:51.638100 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:51.638075    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6922d3-69c86dfc75-ktwbq"]
Oct 30 22:42:51.971658 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:51.965856    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:52.425892 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:52.425849    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5" podUID="74bb46da-f26a-4f88-85e2-c6dc0c305a7a" containerName="payload-processing" containerID="cri-o://5a7c6fa0ec38f2346225df97d9b49421677a2b347d0a5a1d99727f4b6374de7b" gracePeriod=25
Oct 30 22:42:52.777666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:52.777631    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c24e6bd5-4b2f-4060-b4cb-b2df03309b3a" path="/var/lib/kubelet/pods/c24e6bd5-4b2f-4060-b4cb-b2df03309b3a/volumes"
Oct 30 22:42:53.754287 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:53.751712    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:54.989562 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:54.989501    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"]
Oct 30 22:42:54.990511 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:54.990492    4889 cpu_manager.go:414] "containerMap: removing container" podUID="7221740b-e003-4af1-86a2-396e32fff7ca" containerName="istio-proxy"
Oct 30 22:42:54.990580 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:54.990513    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7221740b-e003-4af1-86a2-396e32fff7ca" containerName="istio-proxy"
Oct 30 22:42:54.990580 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:54.990527    4889 cpu_manager.go:414] "containerMap: removing container" podUID="b211a3e6-ec7c-418c-9fa1-a2546cacd7f7" containerName="istio-proxy"
Oct 30 22:42:54.990580 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:54.990531    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b211a3e6-ec7c-418c-9fa1-a2546cacd7f7" containerName="istio-proxy"
Oct 30 22:42:54.990580 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:54.990540    4889 cpu_manager.go:414] "containerMap: removing container" podUID="137329f9-9437-46ab-b030-ead038ccd542" containerName="istio-proxy"
Oct 30 22:42:54.990580 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:54.990546    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="137329f9-9437-46ab-b030-ead038ccd542" containerName="istio-proxy"
Oct 30 22:42:54.990580 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:54.990556    4889 cpu_manager.go:414] "containerMap: removing container" podUID="c24e6bd5-4b2f-4060-b4cb-b2df03309b3a" containerName="payload-pre-processing"
Oct 30 22:42:54.990580 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:54.990560    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c24e6bd5-4b2f-4060-b4cb-b2df03309b3a" containerName="payload-pre-processing"
Oct 30 22:42:54.990821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:54.990610    4889 cpu_manager.go:414] "containerMap: removing container" podUID="8b34996e-28a0-4cbf-b0fa-4615ad9fccaa" containerName="istio-proxy"
Oct 30 22:42:54.990821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:54.990615    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b34996e-28a0-4cbf-b0fa-4615ad9fccaa" containerName="istio-proxy"
Oct 30 22:42:54.990821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:54.990623    4889 cpu_manager.go:414] "containerMap: removing container" podUID="8b34996e-28a0-4cbf-b0fa-4615ad9fccaa" containerName="istio-proxy"
Oct 30 22:42:54.990821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:54.990628    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b34996e-28a0-4cbf-b0fa-4615ad9fccaa" containerName="istio-proxy"
Oct 30 22:42:54.990821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:54.990635    4889 cpu_manager.go:414] "containerMap: removing container" podUID="007bdaf6-3319-49db-850b-7fd9b87a5b7a" containerName="istio-proxy"
Oct 30 22:42:54.990821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:54.990639    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="007bdaf6-3319-49db-850b-7fd9b87a5b7a" containerName="istio-proxy"
Oct 30 22:42:54.990821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:54.990780    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="137329f9-9437-46ab-b030-ead038ccd542" containerName="istio-proxy"
Oct 30 22:42:54.990821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:54.990793    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b34996e-28a0-4cbf-b0fa-4615ad9fccaa" containerName="istio-proxy"
Oct 30 22:42:54.990821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:54.990799    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b34996e-28a0-4cbf-b0fa-4615ad9fccaa" containerName="istio-proxy"
Oct 30 22:42:54.990821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:54.990805    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="007bdaf6-3319-49db-850b-7fd9b87a5b7a" containerName="istio-proxy"
Oct 30 22:42:54.990821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:54.990812    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="7221740b-e003-4af1-86a2-396e32fff7ca" containerName="istio-proxy"
Oct 30 22:42:54.990821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:54.990818    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="7221740b-e003-4af1-86a2-396e32fff7ca" containerName="istio-proxy"
Oct 30 22:42:54.990821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:54.990825    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="b211a3e6-ec7c-418c-9fa1-a2546cacd7f7" containerName="istio-proxy"
Oct 30 22:42:54.991259 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:54.990831    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="137329f9-9437-46ab-b030-ead038ccd542" containerName="istio-proxy"
Oct 30 22:42:54.991259 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:54.990838    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="c24e6bd5-4b2f-4060-b4cb-b2df03309b3a" containerName="payload-pre-processing"
Oct 30 22:42:54.996290 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:54.996270    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.018383 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.018354    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"]
Oct 30 22:42:55.047728 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.031260    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6e0edc-openshift-default-dockercfg-pkdjc\""
Oct 30 22:42:55.049635 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.049589    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istiod-ca-cert\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.049780 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.049706    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-workload-socket\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.049780 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.049739    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvrrc\" (UniqueName: \"kubernetes.io/projected/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-kube-api-access-qvrrc\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.049904 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.049851    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-envoy\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.049904 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.049888    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-token\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.050010 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.049999    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-workload-certs\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.050063 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.050041    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-data\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.050114 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.050100    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-credential-socket\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.050166 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.050149    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-podinfo\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.152120 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.152083    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-workload-certs\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.152320 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.152134    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-data\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.152320 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.152174    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-credential-socket\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.152320 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.152205    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-podinfo\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.152320 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.152290    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istiod-ca-cert\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.152507 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.152381    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-workload-socket\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.152507 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.152409    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvrrc\" (UniqueName: \"kubernetes.io/projected/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-kube-api-access-qvrrc\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.152628 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.152517    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-workload-certs\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.152628 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.152596    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-data\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.152748 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.152668    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-credential-socket\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.152823 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.152797    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-workload-socket\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.152877 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.152845    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-envoy\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.152929 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.152898    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-token\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.153044 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.153024    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istiod-ca-cert\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.155027 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.155004    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-envoy\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.155126 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.155085    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-podinfo\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.182505 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.182476    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvrrc\" (UniqueName: \"kubernetes.io/projected/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-kube-api-access-qvrrc\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.182791 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.182776    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-token\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.312950 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.312915    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:55.668328 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.657950    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:55.679172 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.679137    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"]
Oct 30 22:42:55.685982 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:42:55.685952    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffbcb47e_8cd4_4746_b9b2_ff257d8cde01.slice/crio-3cafad3493c80bf9948988d568988df4a24a0c880d3d17a426d2c09b5fa37ad1 WatchSource:0}: Error finding container 3cafad3493c80bf9948988d568988df4a24a0c880d3d17a426d2c09b5fa37ad1: Status 404 returned error can't find the container with id 3cafad3493c80bf9948988d568988df4a24a0c880d3d17a426d2c09b5fa37ad1
Oct 30 22:42:55.690564 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.690525    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:42:55.690654 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.690611    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:42:55.690698 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.690654    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:42:55.888383 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:55.888341    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5" podUID="74bb46da-f26a-4f88-85e2-c6dc0c305a7a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:42:56.116096 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:56.116047    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:56.638813 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:56.638708    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln" event={"ID":"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01","Type":"ContainerStarted","Data":"37dc7fac3bb494743a71969dbfdb8f9f3fe7782c663a4c20bf85f912070d4f9e"}
Oct 30 22:42:56.638813 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:56.638780    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln" event={"ID":"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01","Type":"ContainerStarted","Data":"3cafad3493c80bf9948988d568988df4a24a0c880d3d17a426d2c09b5fa37ad1"}
Oct 30 22:42:56.736161 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:56.736106    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln" podStartSLOduration=2.736088478 podStartE2EDuration="2.736088478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:56.701771113 +0000 UTC m=+2162.058137428" watchObservedRunningTime="2026-10-30 22:42:56.736088478 +0000 UTC m=+2162.092454793"
Oct 30 22:42:57.331248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:57.316565    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:42:57.335417 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:57.335380    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 22:42:57.335568 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:57.335453    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln" podUID="ffbcb47e-8cd4-4746-b9b2-ff257d8cde01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 22:42:57.670398 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:57.670299    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:58.331248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:58.330586    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 22:42:58.331248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:58.330651    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln" podUID="ffbcb47e-8cd4-4746-b9b2-ff257d8cde01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 22:42:59.333251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:59.332595    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 22:42:59.333251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:59.332672    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln" podUID="ffbcb47e-8cd4-4746-b9b2-ff257d8cde01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 22:42:59.542579 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:42:59.542543    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:00.319251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:00.313527    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 22:43:00.319251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:00.313585    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln" podUID="ffbcb47e-8cd4-4746-b9b2-ff257d8cde01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 22:43:01.334623 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:01.334586    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 22:43:01.335134 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:01.334647    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln" podUID="ffbcb47e-8cd4-4746-b9b2-ff257d8cde01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 22:43:01.580255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:01.577895    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:02.313294 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:02.313218    4889 patch_prober.go:29] interesting pod/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 22:43:02.313479 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:02.313317    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln" podUID="ffbcb47e-8cd4-4746-b9b2-ff257d8cde01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 22:43:03.319726 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:03.319694    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:43:03.320259 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:03.320238    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:43:03.320990 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:03.320969    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:43:03.448343 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:03.441985    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:05.339252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:05.338730    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:05.888899 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:05.888859    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5" podUID="74bb46da-f26a-4f88-85e2-c6dc0c305a7a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:43:07.171672 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:07.169193    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:07.946816 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:07.946780    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:08.116680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:08.116641    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:08.902922 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:08.902828    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:10.303990 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:43:10.303950    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:43:10.671759 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:10.666215    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:12.707303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:12.695748    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:12.779087 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:12.779061    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5"
Oct 30 22:43:12.804843 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:12.804811    4889 generic.go:356] "Generic (PLEG): container finished" podID="74bb46da-f26a-4f88-85e2-c6dc0c305a7a" containerID="5a7c6fa0ec38f2346225df97d9b49421677a2b347d0a5a1d99727f4b6374de7b" exitCode=0
Oct 30 22:43:12.805006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:12.804854    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5"
Oct 30 22:43:12.805006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:12.804898    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5" event={"ID":"74bb46da-f26a-4f88-85e2-c6dc0c305a7a","Type":"ContainerDied","Data":"5a7c6fa0ec38f2346225df97d9b49421677a2b347d0a5a1d99727f4b6374de7b"}
Oct 30 22:43:12.805006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:12.804957    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5" event={"ID":"74bb46da-f26a-4f88-85e2-c6dc0c305a7a","Type":"ContainerDied","Data":"963f3d3015fa2f90468691f49fa283992430996336fd0945ad5faec84eb3e81a"}
Oct 30 22:43:12.805006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:12.804986    4889 scope.go:122] "RemoveContainer" containerID="5a7c6fa0ec38f2346225df97d9b49421677a2b347d0a5a1d99727f4b6374de7b"
Oct 30 22:43:12.816282 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:12.816252    4889 scope.go:122] "RemoveContainer" containerID="5a7c6fa0ec38f2346225df97d9b49421677a2b347d0a5a1d99727f4b6374de7b"
Oct 30 22:43:12.816570 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:43:12.816548    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a7c6fa0ec38f2346225df97d9b49421677a2b347d0a5a1d99727f4b6374de7b\": container with ID starting with 5a7c6fa0ec38f2346225df97d9b49421677a2b347d0a5a1d99727f4b6374de7b not found: ID does not exist" containerID="5a7c6fa0ec38f2346225df97d9b49421677a2b347d0a5a1d99727f4b6374de7b"
Oct 30 22:43:12.816658 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:12.816573    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a7c6fa0ec38f2346225df97d9b49421677a2b347d0a5a1d99727f4b6374de7b"} err="failed to get container status \"5a7c6fa0ec38f2346225df97d9b49421677a2b347d0a5a1d99727f4b6374de7b\": rpc error: code = NotFound desc = could not find container \"5a7c6fa0ec38f2346225df97d9b49421677a2b347d0a5a1d99727f4b6374de7b\": container with ID starting with 5a7c6fa0ec38f2346225df97d9b49421677a2b347d0a5a1d99727f4b6374de7b not found: ID does not exist"
Oct 30 22:43:12.892981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:12.892951    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74bb46da-f26a-4f88-85e2-c6dc0c305a7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74bb46da-f26a-4f88-85e2-c6dc0c305a7a-plugins-config-volume\") pod \"74bb46da-f26a-4f88-85e2-c6dc0c305a7a\" (UID: \"74bb46da-f26a-4f88-85e2-c6dc0c305a7a\") "
Oct 30 22:43:12.893184 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:12.893011    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74bb46da-f26a-4f88-85e2-c6dc0c305a7a-kube-api-access-tlx4b\" (UniqueName: \"kubernetes.io/projected/74bb46da-f26a-4f88-85e2-c6dc0c305a7a-kube-api-access-tlx4b\") pod \"74bb46da-f26a-4f88-85e2-c6dc0c305a7a\" (UID: \"74bb46da-f26a-4f88-85e2-c6dc0c305a7a\") "
Oct 30 22:43:12.893425 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:12.893387    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74bb46da-f26a-4f88-85e2-c6dc0c305a7a-plugins-config-volume" pod "74bb46da-f26a-4f88-85e2-c6dc0c305a7a" (UID: "74bb46da-f26a-4f88-85e2-c6dc0c305a7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:43:12.893900 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:12.893878    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74bb46da-f26a-4f88-85e2-c6dc0c305a7a-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:12.896010 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:12.895977    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74bb46da-f26a-4f88-85e2-c6dc0c305a7a-kube-api-access-tlx4b" pod "74bb46da-f26a-4f88-85e2-c6dc0c305a7a" (UID: "74bb46da-f26a-4f88-85e2-c6dc0c305a7a"). InnerVolumeSpecName "kube-api-access-tlx4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:12.994970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:12.994936    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlx4b\" (UniqueName: \"kubernetes.io/projected/74bb46da-f26a-4f88-85e2-c6dc0c305a7a-kube-api-access-tlx4b\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:13.158445 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:13.158405    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5"]
Oct 30 22:43:13.162645 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:13.162620    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f9d87b-66796c8fff-5mhk5"]
Oct 30 22:43:14.523636 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:14.523604    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:14.777372 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:14.777299    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74bb46da-f26a-4f88-85e2-c6dc0c305a7a" path="/var/lib/kubelet/pods/74bb46da-f26a-4f88-85e2-c6dc0c305a7a/volumes"
Oct 30 22:43:16.139891 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:16.139852    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"]
Oct 30 22:43:16.275179 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:16.275075    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:16.464637 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:16.464518    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:18.363164 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:18.357119    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:20.126106 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.126070    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"]
Oct 30 22:43:20.126549 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.126305    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c" podUID="07bebf45-5427-4745-bbb8-6ec51bd54042" containerName="discovery" containerID="cri-o://d3b878eac4db8b9ec3e40abaae4bedf09f667d4270ccdd808a9eb91bab559b3a" gracePeriod=30
Oct 30 22:43:20.166035 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.165916    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:20.442267 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.442207    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"
Oct 30 22:43:20.578736 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.578701    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07bebf45-5427-4745-bbb8-6ec51bd54042-cacerts\" (UniqueName: \"kubernetes.io/secret/07bebf45-5427-4745-bbb8-6ec51bd54042-cacerts\") pod \"07bebf45-5427-4745-bbb8-6ec51bd54042\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") "
Oct 30 22:43:20.578939 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.578754    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-csr-ca-configmap\") pod \"07bebf45-5427-4745-bbb8-6ec51bd54042\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") "
Oct 30 22:43:20.578939 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.578795    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-kubeconfig\") pod \"07bebf45-5427-4745-bbb8-6ec51bd54042\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") "
Oct 30 22:43:20.579036 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.578954    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07bebf45-5427-4745-bbb8-6ec51bd54042-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/07bebf45-5427-4745-bbb8-6ec51bd54042-local-certs\") pod \"07bebf45-5427-4745-bbb8-6ec51bd54042\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") "
Oct 30 22:43:20.579102 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.579053    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-token\" (UniqueName: \"kubernetes.io/projected/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-token\") pod \"07bebf45-5427-4745-bbb8-6ec51bd54042\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") "
Oct 30 22:43:20.579205 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.579177    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-csr-ca-configmap" pod "07bebf45-5427-4745-bbb8-6ec51bd54042" (UID: "07bebf45-5427-4745-bbb8-6ec51bd54042"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:43:20.579205 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.579195    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-csr-dns-cert\") pod \"07bebf45-5427-4745-bbb8-6ec51bd54042\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") "
Oct 30 22:43:20.579436 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.579401    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07bebf45-5427-4745-bbb8-6ec51bd54042-kube-api-access-9tk25\" (UniqueName: \"kubernetes.io/projected/07bebf45-5427-4745-bbb8-6ec51bd54042-kube-api-access-9tk25\") pod \"07bebf45-5427-4745-bbb8-6ec51bd54042\" (UID: \"07bebf45-5427-4745-bbb8-6ec51bd54042\") "
Oct 30 22:43:20.580447 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.580420    4889 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-csr-ca-configmap\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:20.581830 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.581799    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-token" pod "07bebf45-5427-4745-bbb8-6ec51bd54042" (UID: "07bebf45-5427-4745-bbb8-6ec51bd54042"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:20.581986 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.581821    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bebf45-5427-4745-bbb8-6ec51bd54042-local-certs" pod "07bebf45-5427-4745-bbb8-6ec51bd54042" (UID: "07bebf45-5427-4745-bbb8-6ec51bd54042"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:20.582056 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.581986    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-kubeconfig" pod "07bebf45-5427-4745-bbb8-6ec51bd54042" (UID: "07bebf45-5427-4745-bbb8-6ec51bd54042"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:43:20.582056 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.582007    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07bebf45-5427-4745-bbb8-6ec51bd54042-cacerts" pod "07bebf45-5427-4745-bbb8-6ec51bd54042" (UID: "07bebf45-5427-4745-bbb8-6ec51bd54042"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:43:20.582632 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.582598    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07bebf45-5427-4745-bbb8-6ec51bd54042-kube-api-access-9tk25" pod "07bebf45-5427-4745-bbb8-6ec51bd54042" (UID: "07bebf45-5427-4745-bbb8-6ec51bd54042"). InnerVolumeSpecName "kube-api-access-9tk25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:20.584017 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.583990    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-csr-dns-cert" pod "07bebf45-5427-4745-bbb8-6ec51bd54042" (UID: "07bebf45-5427-4745-bbb8-6ec51bd54042"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:43:20.682407 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.682314    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tk25\" (UniqueName: \"kubernetes.io/projected/07bebf45-5427-4745-bbb8-6ec51bd54042-kube-api-access-9tk25\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:20.682407 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.682348    4889 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/07bebf45-5427-4745-bbb8-6ec51bd54042-cacerts\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:20.682407 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.682361    4889 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-kubeconfig\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:20.682407 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.682375    4889 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/07bebf45-5427-4745-bbb8-6ec51bd54042-local-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:20.682407 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.682386    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:20.682407 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.682397    4889 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/07bebf45-5427-4745-bbb8-6ec51bd54042-istio-csr-dns-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:20.879029 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.878993    4889 generic.go:356] "Generic (PLEG): container finished" podID="07bebf45-5427-4745-bbb8-6ec51bd54042" containerID="d3b878eac4db8b9ec3e40abaae4bedf09f667d4270ccdd808a9eb91bab559b3a" exitCode=0
Oct 30 22:43:20.879204 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.879040    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c" event={"ID":"07bebf45-5427-4745-bbb8-6ec51bd54042","Type":"ContainerDied","Data":"d3b878eac4db8b9ec3e40abaae4bedf09f667d4270ccdd808a9eb91bab559b3a"}
Oct 30 22:43:20.879204 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.879075    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c" event={"ID":"07bebf45-5427-4745-bbb8-6ec51bd54042","Type":"ContainerDied","Data":"531404b9de387a65c4b63e61e3e01aaf8da23c29ebacf66df39f430d1f0e9b8d"}
Oct 30 22:43:20.879204 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.879095    4889 scope.go:122] "RemoveContainer" containerID="d3b878eac4db8b9ec3e40abaae4bedf09f667d4270ccdd808a9eb91bab559b3a"
Oct 30 22:43:20.879337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.879217    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"
Oct 30 22:43:20.898998 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.898973    4889 scope.go:122] "RemoveContainer" containerID="d3b878eac4db8b9ec3e40abaae4bedf09f667d4270ccdd808a9eb91bab559b3a"
Oct 30 22:43:20.899314 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:43:20.899292    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3b878eac4db8b9ec3e40abaae4bedf09f667d4270ccdd808a9eb91bab559b3a\": container with ID starting with d3b878eac4db8b9ec3e40abaae4bedf09f667d4270ccdd808a9eb91bab559b3a not found: ID does not exist" containerID="d3b878eac4db8b9ec3e40abaae4bedf09f667d4270ccdd808a9eb91bab559b3a"
Oct 30 22:43:20.899416 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.899323    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3b878eac4db8b9ec3e40abaae4bedf09f667d4270ccdd808a9eb91bab559b3a"} err="failed to get container status \"d3b878eac4db8b9ec3e40abaae4bedf09f667d4270ccdd808a9eb91bab559b3a\": rpc error: code = NotFound desc = could not find container \"d3b878eac4db8b9ec3e40abaae4bedf09f667d4270ccdd808a9eb91bab559b3a\": container with ID starting with d3b878eac4db8b9ec3e40abaae4bedf09f667d4270ccdd808a9eb91bab559b3a not found: ID does not exist"
Oct 30 22:43:20.951982 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.951900    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"]
Oct 30 22:43:20.958282 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:20.958252    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c926c"]
Oct 30 22:43:22.016474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:22.016437    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:22.777201 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:22.777166    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07bebf45-5427-4745-bbb8-6ec51bd54042" path="/var/lib/kubelet/pods/07bebf45-5427-4745-bbb8-6ec51bd54042/volumes"
Oct 30 22:43:23.986058 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:23.986019    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:25.808727 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:25.808677    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:27.112336 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:27.112298    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:27.280692 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:27.280655    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:27.652956 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:27.652914    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:29.583068 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:29.583007    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:31.163557 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:31.163506    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl" podUID="d863aec2-a41e-44bb-a09d-856643916b65" containerName="main" containerID="cri-o://abc9f778764a4fed983e27ac911d9d7edefa3ab49702d7c5a15c666112cd8ed6" gracePeriod=45
Oct 30 22:43:31.316978 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:31.316938    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:31.516710 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:31.516669    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"
Oct 30 22:43:31.574813 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:31.574756    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d863aec2-a41e-44bb-a09d-856643916b65-kube-api-access-lppqf\" (UniqueName: \"kubernetes.io/projected/d863aec2-a41e-44bb-a09d-856643916b65-kube-api-access-lppqf\") pod \"d863aec2-a41e-44bb-a09d-856643916b65\" (UID: \"d863aec2-a41e-44bb-a09d-856643916b65\") "
Oct 30 22:43:31.574997 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:31.574934    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-dshm\") pod \"d863aec2-a41e-44bb-a09d-856643916b65\" (UID: \"d863aec2-a41e-44bb-a09d-856643916b65\") "
Oct 30 22:43:31.575042 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:31.574996    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-home\" (UniqueName: \"kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-home\") pod \"d863aec2-a41e-44bb-a09d-856643916b65\" (UID: \"d863aec2-a41e-44bb-a09d-856643916b65\") "
Oct 30 22:43:31.575191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:31.575172    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d863aec2-a41e-44bb-a09d-856643916b65-tls-certs\" (UniqueName: \"kubernetes.io/secret/d863aec2-a41e-44bb-a09d-856643916b65-tls-certs\") pod \"d863aec2-a41e-44bb-a09d-856643916b65\" (UID: \"d863aec2-a41e-44bb-a09d-856643916b65\") "
Oct 30 22:43:31.575258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:31.575244    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-model-cache\") pod \"d863aec2-a41e-44bb-a09d-856643916b65\" (UID: \"d863aec2-a41e-44bb-a09d-856643916b65\") "
Oct 30 22:43:31.575370 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:31.575355    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-tmp-dir\") pod \"d863aec2-a41e-44bb-a09d-856643916b65\" (UID: \"d863aec2-a41e-44bb-a09d-856643916b65\") "
Oct 30 22:43:31.577001 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:31.576968    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-tmp-dir" pod "d863aec2-a41e-44bb-a09d-856643916b65" (UID: "d863aec2-a41e-44bb-a09d-856643916b65"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:31.577397 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:31.577367    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-model-cache" pod "d863aec2-a41e-44bb-a09d-856643916b65" (UID: "d863aec2-a41e-44bb-a09d-856643916b65"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:31.577633 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:31.577610    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-home" pod "d863aec2-a41e-44bb-a09d-856643916b65" (UID: "d863aec2-a41e-44bb-a09d-856643916b65"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:31.579523 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:31.579470    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d863aec2-a41e-44bb-a09d-856643916b65-tls-certs" pod "d863aec2-a41e-44bb-a09d-856643916b65" (UID: "d863aec2-a41e-44bb-a09d-856643916b65"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:43:31.579523 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:31.579476    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-dshm" pod "d863aec2-a41e-44bb-a09d-856643916b65" (UID: "d863aec2-a41e-44bb-a09d-856643916b65"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:31.579851 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:31.579808    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d863aec2-a41e-44bb-a09d-856643916b65-kube-api-access-lppqf" pod "d863aec2-a41e-44bb-a09d-856643916b65" (UID: "d863aec2-a41e-44bb-a09d-856643916b65"). InnerVolumeSpecName "kube-api-access-lppqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:31.678550 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:31.678511    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lppqf\" (UniqueName: \"kubernetes.io/projected/d863aec2-a41e-44bb-a09d-856643916b65-kube-api-access-lppqf\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:31.678550 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:31.678542    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:31.678550 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:31.678551    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:31.678550 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:31.678559    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d863aec2-a41e-44bb-a09d-856643916b65-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:31.678863 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:31.678568    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:31.678863 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:31.678575    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d863aec2-a41e-44bb-a09d-856643916b65-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:32.027028 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:32.026980    4889 generic.go:356] "Generic (PLEG): container finished" podID="d863aec2-a41e-44bb-a09d-856643916b65" containerID="abc9f778764a4fed983e27ac911d9d7edefa3ab49702d7c5a15c666112cd8ed6" exitCode=0
Oct 30 22:43:32.027242 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:32.027082    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl" event={"ID":"d863aec2-a41e-44bb-a09d-856643916b65","Type":"ContainerDied","Data":"abc9f778764a4fed983e27ac911d9d7edefa3ab49702d7c5a15c666112cd8ed6"}
Oct 30 22:43:32.027242 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:32.027103    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"
Oct 30 22:43:32.027242 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:32.027138    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl" event={"ID":"d863aec2-a41e-44bb-a09d-856643916b65","Type":"ContainerDied","Data":"f4a1ddb45d9f0c32427e8d21a19e4af4f023f1c4af07cac3426714fd249d7e4e"}
Oct 30 22:43:32.027242 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:32.027160    4889 scope.go:122] "RemoveContainer" containerID="abc9f778764a4fed983e27ac911d9d7edefa3ab49702d7c5a15c666112cd8ed6"
Oct 30 22:43:32.041826 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:32.041801    4889 scope.go:122] "RemoveContainer" containerID="abc9f778764a4fed983e27ac911d9d7edefa3ab49702d7c5a15c666112cd8ed6"
Oct 30 22:43:32.042319 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:43:32.042290    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abc9f778764a4fed983e27ac911d9d7edefa3ab49702d7c5a15c666112cd8ed6\": container with ID starting with abc9f778764a4fed983e27ac911d9d7edefa3ab49702d7c5a15c666112cd8ed6 not found: ID does not exist" containerID="abc9f778764a4fed983e27ac911d9d7edefa3ab49702d7c5a15c666112cd8ed6"
Oct 30 22:43:32.042443 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:32.042327    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abc9f778764a4fed983e27ac911d9d7edefa3ab49702d7c5a15c666112cd8ed6"} err="failed to get container status \"abc9f778764a4fed983e27ac911d9d7edefa3ab49702d7c5a15c666112cd8ed6\": rpc error: code = NotFound desc = could not find container \"abc9f778764a4fed983e27ac911d9d7edefa3ab49702d7c5a15c666112cd8ed6\": container with ID starting with abc9f778764a4fed983e27ac911d9d7edefa3ab49702d7c5a15c666112cd8ed6 not found: ID does not exist"
Oct 30 22:43:32.087523 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:32.087457    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"]
Oct 30 22:43:32.107378 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:32.107297    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-6013c8/test-model-6013c8-kserve-85fcbf9bc7-chrjl"]
Oct 30 22:43:32.777852 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:32.777818    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d863aec2-a41e-44bb-a09d-856643916b65" path="/var/lib/kubelet/pods/d863aec2-a41e-44bb-a09d-856643916b65/volumes"
Oct 30 22:43:33.142283 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:33.142132    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:34.490266 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:34.490215    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c"]
Oct 30 22:43:34.490770 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:34.490497    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c" podUID="625280d4-8b36-4188-a38d-725f4cdd55e3" containerName="maas-api" containerID="cri-o://cbac7be9e76cf849f6e2ca31a3d5a6c2d8ca7b8ee32b609a5b454859cd9aeb7a" gracePeriod=30
Oct 30 22:43:34.616206 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:34.616168    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:34.860199 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:34.860167    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c"
Oct 30 22:43:34.957630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:34.954954    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/625280d4-8b36-4188-a38d-725f4cdd55e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/625280d4-8b36-4188-a38d-725f4cdd55e3-openshift-service-ca\") pod \"625280d4-8b36-4188-a38d-725f4cdd55e3\" (UID: \"625280d4-8b36-4188-a38d-725f4cdd55e3\") "
Oct 30 22:43:34.957630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:34.955147    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/625280d4-8b36-4188-a38d-725f4cdd55e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625280d4-8b36-4188-a38d-725f4cdd55e3-maas-api-tls\") pod \"625280d4-8b36-4188-a38d-725f4cdd55e3\" (UID: \"625280d4-8b36-4188-a38d-725f4cdd55e3\") "
Oct 30 22:43:34.957630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:34.955251    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/625280d4-8b36-4188-a38d-725f4cdd55e3-kube-api-access-nkkb5\" (UniqueName: \"kubernetes.io/projected/625280d4-8b36-4188-a38d-725f4cdd55e3-kube-api-access-nkkb5\") pod \"625280d4-8b36-4188-a38d-725f4cdd55e3\" (UID: \"625280d4-8b36-4188-a38d-725f4cdd55e3\") "
Oct 30 22:43:34.957630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:34.955299    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/625280d4-8b36-4188-a38d-725f4cdd55e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/625280d4-8b36-4188-a38d-725f4cdd55e3-metrics-certs\") pod \"625280d4-8b36-4188-a38d-725f4cdd55e3\" (UID: \"625280d4-8b36-4188-a38d-725f4cdd55e3\") "
Oct 30 22:43:34.957630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:34.957356    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625280d4-8b36-4188-a38d-725f4cdd55e3-openshift-service-ca" pod "625280d4-8b36-4188-a38d-725f4cdd55e3" (UID: "625280d4-8b36-4188-a38d-725f4cdd55e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:43:35.007053 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:35.006828    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/625280d4-8b36-4188-a38d-725f4cdd55e3-maas-api-tls" pod "625280d4-8b36-4188-a38d-725f4cdd55e3" (UID: "625280d4-8b36-4188-a38d-725f4cdd55e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:43:35.008448 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:35.008394    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625280d4-8b36-4188-a38d-725f4cdd55e3-kube-api-access-nkkb5" pod "625280d4-8b36-4188-a38d-725f4cdd55e3" (UID: "625280d4-8b36-4188-a38d-725f4cdd55e3"). InnerVolumeSpecName "kube-api-access-nkkb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:35.023315 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:35.008699    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/625280d4-8b36-4188-a38d-725f4cdd55e3-metrics-certs" pod "625280d4-8b36-4188-a38d-725f4cdd55e3" (UID: "625280d4-8b36-4188-a38d-725f4cdd55e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:43:35.069661 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:35.069477    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625280d4-8b36-4188-a38d-725f4cdd55e3-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:35.069661 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:35.069510    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkkb5\" (UniqueName: \"kubernetes.io/projected/625280d4-8b36-4188-a38d-725f4cdd55e3-kube-api-access-nkkb5\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:35.069661 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:35.069529    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/625280d4-8b36-4188-a38d-725f4cdd55e3-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:35.069661 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:35.069547    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/625280d4-8b36-4188-a38d-725f4cdd55e3-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:35.109871 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:35.107934    4889 generic.go:356] "Generic (PLEG): container finished" podID="625280d4-8b36-4188-a38d-725f4cdd55e3" containerID="cbac7be9e76cf849f6e2ca31a3d5a6c2d8ca7b8ee32b609a5b454859cd9aeb7a" exitCode=0
Oct 30 22:43:35.109871 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:35.108009    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c" event={"ID":"625280d4-8b36-4188-a38d-725f4cdd55e3","Type":"ContainerDied","Data":"cbac7be9e76cf849f6e2ca31a3d5a6c2d8ca7b8ee32b609a5b454859cd9aeb7a"}
Oct 30 22:43:35.109871 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:35.108059    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c" event={"ID":"625280d4-8b36-4188-a38d-725f4cdd55e3","Type":"ContainerDied","Data":"c5d4991c73787b440e823099dd1b36e9532227120c6aaeaad24719db8da9daeb"}
Oct 30 22:43:35.109871 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:35.108093    4889 scope.go:122] "RemoveContainer" containerID="cbac7be9e76cf849f6e2ca31a3d5a6c2d8ca7b8ee32b609a5b454859cd9aeb7a"
Oct 30 22:43:35.109871 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:35.108213    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c"
Oct 30 22:43:35.147650 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:35.147609    4889 scope.go:122] "RemoveContainer" containerID="cbac7be9e76cf849f6e2ca31a3d5a6c2d8ca7b8ee32b609a5b454859cd9aeb7a"
Oct 30 22:43:35.148077 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:43:35.148051    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbac7be9e76cf849f6e2ca31a3d5a6c2d8ca7b8ee32b609a5b454859cd9aeb7a\": container with ID starting with cbac7be9e76cf849f6e2ca31a3d5a6c2d8ca7b8ee32b609a5b454859cd9aeb7a not found: ID does not exist" containerID="cbac7be9e76cf849f6e2ca31a3d5a6c2d8ca7b8ee32b609a5b454859cd9aeb7a"
Oct 30 22:43:35.148193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:35.148089    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbac7be9e76cf849f6e2ca31a3d5a6c2d8ca7b8ee32b609a5b454859cd9aeb7a"} err="failed to get container status \"cbac7be9e76cf849f6e2ca31a3d5a6c2d8ca7b8ee32b609a5b454859cd9aeb7a\": rpc error: code = NotFound desc = could not find container \"cbac7be9e76cf849f6e2ca31a3d5a6c2d8ca7b8ee32b609a5b454859cd9aeb7a\": container with ID starting with cbac7be9e76cf849f6e2ca31a3d5a6c2d8ca7b8ee32b609a5b454859cd9aeb7a not found: ID does not exist"
Oct 30 22:43:35.182483 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:35.182452    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c"]
Oct 30 22:43:35.189247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:35.189195    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f9d87b-8db56965f-v4m7c"]
Oct 30 22:43:35.282532 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:35.282493    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:35.451802 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:35.451704    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:35.968645 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:35.968593    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:36.779218 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:36.779181    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="625280d4-8b36-4188-a38d-725f4cdd55e3" path="/var/lib/kubelet/pods/625280d4-8b36-4188-a38d-725f4cdd55e3/volumes"
Oct 30 22:43:37.369476 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:37.369437    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:39.325641 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:39.325598    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:40.305857 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:43:40.305818    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:43:41.146197 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:41.146149    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:42.744941 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:42.744905    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:42.928507 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:42.928472    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:43.011679 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.011577    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"]
Oct 30 22:43:43.011881 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.011781    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk" podUID="7b49c0bf-aa2d-4f17-8a09-831d9b8bc459" containerName="istio-proxy" containerID="cri-o://4429583f8ee29228a72cea60ac51ff8eafcccbf17f65a9b42882aca907b44044" gracePeriod=30
Oct 30 22:43:43.148257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.148140    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:43.630320 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.630284    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"]
Oct 30 22:43:43.631163 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.631148    4889 cpu_manager.go:414] "containerMap: removing container" podUID="74bb46da-f26a-4f88-85e2-c6dc0c305a7a" containerName="payload-processing"
Oct 30 22:43:43.631215 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.631165    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74bb46da-f26a-4f88-85e2-c6dc0c305a7a" containerName="payload-processing"
Oct 30 22:43:43.631215 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.631177    4889 cpu_manager.go:414] "containerMap: removing container" podUID="d863aec2-a41e-44bb-a09d-856643916b65" containerName="main"
Oct 30 22:43:43.631215 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.631182    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d863aec2-a41e-44bb-a09d-856643916b65" containerName="main"
Oct 30 22:43:43.631215 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.631213    4889 cpu_manager.go:414] "containerMap: removing container" podUID="625280d4-8b36-4188-a38d-725f4cdd55e3" containerName="maas-api"
Oct 30 22:43:43.631360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.631232    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="625280d4-8b36-4188-a38d-725f4cdd55e3" containerName="maas-api"
Oct 30 22:43:43.631360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.631249    4889 cpu_manager.go:414] "containerMap: removing container" podUID="07bebf45-5427-4745-bbb8-6ec51bd54042" containerName="discovery"
Oct 30 22:43:43.631360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.631254    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07bebf45-5427-4745-bbb8-6ec51bd54042" containerName="discovery"
Oct 30 22:43:43.631360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.631271    4889 cpu_manager.go:414] "containerMap: removing container" podUID="7221740b-e003-4af1-86a2-396e32fff7ca" containerName="istio-proxy"
Oct 30 22:43:43.631360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.631275    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7221740b-e003-4af1-86a2-396e32fff7ca" containerName="istio-proxy"
Oct 30 22:43:43.631360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.631285    4889 cpu_manager.go:414] "containerMap: removing container" podUID="137329f9-9437-46ab-b030-ead038ccd542" containerName="istio-proxy"
Oct 30 22:43:43.631360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.631289    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="137329f9-9437-46ab-b030-ead038ccd542" containerName="istio-proxy"
Oct 30 22:43:43.631980 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.631955    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="625280d4-8b36-4188-a38d-725f4cdd55e3" containerName="maas-api"
Oct 30 22:43:43.632094 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.632012    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="07bebf45-5427-4745-bbb8-6ec51bd54042" containerName="discovery"
Oct 30 22:43:43.632094 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.632047    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="74bb46da-f26a-4f88-85e2-c6dc0c305a7a" containerName="payload-processing"
Oct 30 22:43:43.632094 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.632066    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="d863aec2-a41e-44bb-a09d-856643916b65" containerName="main"
Oct 30 22:43:43.644251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.642187    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.660523 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.660486    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"]
Oct 30 22:43:43.692627 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.692584    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-40229e-openshift-default-dockercfg-7dwpw\""
Oct 30 22:43:43.722643 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.722596    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-workload-socket\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.722860 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.722659    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74vqc\" (UniqueName: \"kubernetes.io/projected/e2029821-7bca-434d-978b-07c37e50d52b-kube-api-access-74vqc\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.722860 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.722815    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2029821-7bca-434d-978b-07c37e50d52b-istio-token\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.722993 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.722895    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-workload-certs\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.723058 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.723000    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2029821-7bca-434d-978b-07c37e50d52b-istiod-ca-cert\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.723112 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.723102    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-istio-envoy\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.723265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.723234    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-istio-data\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.723331 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.723275    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-credential-socket\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.723331 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.723298    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2029821-7bca-434d-978b-07c37e50d52b-istio-podinfo\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.825154 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.825103    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2029821-7bca-434d-978b-07c37e50d52b-istio-token\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.826627 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.825182    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-workload-certs\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.826627 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.825410    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2029821-7bca-434d-978b-07c37e50d52b-istiod-ca-cert\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.826627 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.825520    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-istio-envoy\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.826627 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.825614    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-workload-certs\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.826627 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.825630    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-istio-data\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.826627 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.825696    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-credential-socket\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.826627 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.825744    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2029821-7bca-434d-978b-07c37e50d52b-istio-podinfo\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.826627 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.825914    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-workload-socket\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.826627 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.826005    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74vqc\" (UniqueName: \"kubernetes.io/projected/e2029821-7bca-434d-978b-07c37e50d52b-kube-api-access-74vqc\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.826627 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.826071    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-credential-socket\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.826627 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.826322    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-workload-socket\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.826627 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.826388    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2029821-7bca-434d-978b-07c37e50d52b-istiod-ca-cert\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.826627 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.826519    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-istio-data\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.828423 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.828397    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-istio-envoy\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.828661 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.828642    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2029821-7bca-434d-978b-07c37e50d52b-istio-podinfo\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.846944 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.846904    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2029821-7bca-434d-978b-07c37e50d52b-istio-token\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.847420 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.847400    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74vqc\" (UniqueName: \"kubernetes.io/projected/e2029821-7bca-434d-978b-07c37e50d52b-kube-api-access-74vqc\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:43.963427 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:43.963332    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:44.302646 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:43:44.302611    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2029821_7bca_434d_978b_07c37e50d52b.slice/crio-d5644bf21076a40eba05c85f8bed879dc650261fbb8919c3fdd825d8a6428350 WatchSource:0}: Error finding container d5644bf21076a40eba05c85f8bed879dc650261fbb8919c3fdd825d8a6428350: Status 404 returned error can't find the container with id d5644bf21076a40eba05c85f8bed879dc650261fbb8919c3fdd825d8a6428350
Oct 30 22:43:44.302902 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:44.302881    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"]
Oct 30 22:43:44.305813 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:44.305770    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:43:44.305943 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:44.305858    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:43:44.305943 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:44.305897    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:43:44.725291 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:44.725123    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:45.215148 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:45.214800    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v" event={"ID":"e2029821-7bca-434d-978b-07c37e50d52b","Type":"ContainerStarted","Data":"b52e48f34bd4d7bffe5e11b4c66c107b9157471409c9566bca7675717636caf6"}
Oct 30 22:43:45.215148 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:45.214877    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v" event={"ID":"e2029821-7bca-434d-978b-07c37e50d52b","Type":"ContainerStarted","Data":"d5644bf21076a40eba05c85f8bed879dc650261fbb8919c3fdd825d8a6428350"}
Oct 30 22:43:45.281326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:45.281268    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v" podStartSLOduration=2.281250198 podStartE2EDuration="2.281250198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:45.277910788 +0000 UTC m=+2210.634277104" watchObservedRunningTime="2026-10-30 22:43:45.281250198 +0000 UTC m=+2210.637616512"
Oct 30 22:43:45.963791 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:45.963755    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:45.965433 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:45.965407    4889 patch_prober.go:29] interesting pod/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:43:45.965575 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:45.965455    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v" podUID="e2029821-7bca-434d-978b-07c37e50d52b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 22:43:46.069183 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:46.069141    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl"]
Oct 30 22:43:46.069466 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:46.069432    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl" podUID="9fe49ab4-341a-4c8a-b993-6885741c01ae" containerName="maas-api" containerID="cri-o://76b84a5c7a008678f5a8a35d499897af7a9d1fbd3cfd120547c1cba95ff488ed" gracePeriod=30
Oct 30 22:43:46.187807 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:46.187763    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:46.258604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:46.258550    4889 generic.go:356] "Generic (PLEG): container finished" podID="9fe49ab4-341a-4c8a-b993-6885741c01ae" containerID="76b84a5c7a008678f5a8a35d499897af7a9d1fbd3cfd120547c1cba95ff488ed" exitCode=0
Oct 30 22:43:46.259119 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:46.258623    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl" event={"ID":"9fe49ab4-341a-4c8a-b993-6885741c01ae","Type":"ContainerDied","Data":"76b84a5c7a008678f5a8a35d499897af7a9d1fbd3cfd120547c1cba95ff488ed"}
Oct 30 22:43:46.714374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:46.710479    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl"
Oct 30 22:43:46.837315 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:46.836557    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fe49ab4-341a-4c8a-b993-6885741c01ae-kube-api-access-5ph22\" (UniqueName: \"kubernetes.io/projected/9fe49ab4-341a-4c8a-b993-6885741c01ae-kube-api-access-5ph22\") pod \"9fe49ab4-341a-4c8a-b993-6885741c01ae\" (UID: \"9fe49ab4-341a-4c8a-b993-6885741c01ae\") "
Oct 30 22:43:46.837315 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:46.836676    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fe49ab4-341a-4c8a-b993-6885741c01ae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fe49ab4-341a-4c8a-b993-6885741c01ae-metrics-certs\") pod \"9fe49ab4-341a-4c8a-b993-6885741c01ae\" (UID: \"9fe49ab4-341a-4c8a-b993-6885741c01ae\") "
Oct 30 22:43:46.837315 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:46.836822    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fe49ab4-341a-4c8a-b993-6885741c01ae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fe49ab4-341a-4c8a-b993-6885741c01ae-maas-api-tls\") pod \"9fe49ab4-341a-4c8a-b993-6885741c01ae\" (UID: \"9fe49ab4-341a-4c8a-b993-6885741c01ae\") "
Oct 30 22:43:46.837315 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:46.836914    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fe49ab4-341a-4c8a-b993-6885741c01ae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe49ab4-341a-4c8a-b993-6885741c01ae-openshift-service-ca\") pod \"9fe49ab4-341a-4c8a-b993-6885741c01ae\" (UID: \"9fe49ab4-341a-4c8a-b993-6885741c01ae\") "
Oct 30 22:43:46.838286 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:46.838194    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe49ab4-341a-4c8a-b993-6885741c01ae-openshift-service-ca" pod "9fe49ab4-341a-4c8a-b993-6885741c01ae" (UID: "9fe49ab4-341a-4c8a-b993-6885741c01ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:43:46.840964 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:46.840930    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fe49ab4-341a-4c8a-b993-6885741c01ae-metrics-certs" pod "9fe49ab4-341a-4c8a-b993-6885741c01ae" (UID: "9fe49ab4-341a-4c8a-b993-6885741c01ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:43:46.841424 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:46.841397    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fe49ab4-341a-4c8a-b993-6885741c01ae-maas-api-tls" pod "9fe49ab4-341a-4c8a-b993-6885741c01ae" (UID: "9fe49ab4-341a-4c8a-b993-6885741c01ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:43:46.842708 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:46.842684    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe49ab4-341a-4c8a-b993-6885741c01ae-kube-api-access-5ph22" pod "9fe49ab4-341a-4c8a-b993-6885741c01ae" (UID: "9fe49ab4-341a-4c8a-b993-6885741c01ae"). InnerVolumeSpecName "kube-api-access-5ph22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:46.906517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:46.906474    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:46.939661 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:46.939624    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ph22\" (UniqueName: \"kubernetes.io/projected/9fe49ab4-341a-4c8a-b993-6885741c01ae-kube-api-access-5ph22\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:46.939661 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:46.939661    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fe49ab4-341a-4c8a-b993-6885741c01ae-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:46.939661 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:46.939671    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fe49ab4-341a-4c8a-b993-6885741c01ae-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:46.939945 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:46.939680    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe49ab4-341a-4c8a-b993-6885741c01ae-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:46.965480 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:46.965438    4889 patch_prober.go:29] interesting pod/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:43:46.965671 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:46.965504    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v" podUID="e2029821-7bca-434d-978b-07c37e50d52b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 22:43:47.084705 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:47.084659    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:47.261497 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:47.261456    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:47.269894 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:47.269859    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl"
Oct 30 22:43:47.269894 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:47.269874    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl" event={"ID":"9fe49ab4-341a-4c8a-b993-6885741c01ae","Type":"ContainerDied","Data":"39f7036807225737d82070f495ca899eb597b29208e0969de60f1089120d48df"}
Oct 30 22:43:47.270142 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:47.269945    4889 scope.go:122] "RemoveContainer" containerID="76b84a5c7a008678f5a8a35d499897af7a9d1fbd3cfd120547c1cba95ff488ed"
Oct 30 22:43:47.344205 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:47.344162    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl"]
Oct 30 22:43:47.356797 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:47.356709    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bdd94b-645b649679-4hcnl"]
Oct 30 22:43:47.582493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:47.582456    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:47.963688 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:47.963655    4889 patch_prober.go:29] interesting pod/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:43:47.963879 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:47.963707    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v" podUID="e2029821-7bca-434d-978b-07c37e50d52b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 22:43:48.284689 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.284651    4889 generic.go:356] "Generic (PLEG): container finished" podID="7b49c0bf-aa2d-4f17-8a09-831d9b8bc459" containerID="4429583f8ee29228a72cea60ac51ff8eafcccbf17f65a9b42882aca907b44044" exitCode=0
Oct 30 22:43:48.285132 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.284874    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk" event={"ID":"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459","Type":"ContainerDied","Data":"4429583f8ee29228a72cea60ac51ff8eafcccbf17f65a9b42882aca907b44044"}
Oct 30 22:43:48.285132 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.284933    4889 scope.go:122] "RemoveContainer" containerID="bf0cb469820dc9eb0b5a8ed43d63125eaa4f4e1a0bd2199e54f55889227bc390"
Oct 30 22:43:48.441969 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.441939    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:43:48.475374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.475273    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-podinfo\") pod \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") "
Oct 30 22:43:48.475530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.475425    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istiod-ca-cert\") pod \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") "
Oct 30 22:43:48.475530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.475456    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-envoy\") pod \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") "
Oct 30 22:43:48.475530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.475494    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-credential-socket\") pod \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") "
Oct 30 22:43:48.475530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.475515    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-workload-certs\") pod \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") "
Oct 30 22:43:48.475758 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.475577    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-workload-socket\") pod \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") "
Oct 30 22:43:48.475758 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.475660    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-kube-api-access-lc6lp\" (UniqueName: \"kubernetes.io/projected/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-kube-api-access-lc6lp\") pod \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") "
Oct 30 22:43:48.475758 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.475745    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-data\") pod \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") "
Oct 30 22:43:48.475924 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.475800    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-token\" (UniqueName: \"kubernetes.io/projected/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-token\") pod \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\" (UID: \"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459\") "
Oct 30 22:43:48.475924 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.475802    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-credential-socket" pod "7b49c0bf-aa2d-4f17-8a09-831d9b8bc459" (UID: "7b49c0bf-aa2d-4f17-8a09-831d9b8bc459"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:48.475924 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.475892    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-workload-certs" pod "7b49c0bf-aa2d-4f17-8a09-831d9b8bc459" (UID: "7b49c0bf-aa2d-4f17-8a09-831d9b8bc459"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:48.476084 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.475924    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-workload-socket" pod "7b49c0bf-aa2d-4f17-8a09-831d9b8bc459" (UID: "7b49c0bf-aa2d-4f17-8a09-831d9b8bc459"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:48.476084 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.475967    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istiod-ca-cert" pod "7b49c0bf-aa2d-4f17-8a09-831d9b8bc459" (UID: "7b49c0bf-aa2d-4f17-8a09-831d9b8bc459"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:43:48.476311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.476103    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-data" pod "7b49c0bf-aa2d-4f17-8a09-831d9b8bc459" (UID: "7b49c0bf-aa2d-4f17-8a09-831d9b8bc459"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:48.476582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.476557    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:48.476733 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.476586    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:48.476733 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.476612    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:48.476869 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.476702    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:48.476869 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.476799    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:48.477936 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.477909    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-envoy" pod "7b49c0bf-aa2d-4f17-8a09-831d9b8bc459" (UID: "7b49c0bf-aa2d-4f17-8a09-831d9b8bc459"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:48.478111 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.478096    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-podinfo" pod "7b49c0bf-aa2d-4f17-8a09-831d9b8bc459" (UID: "7b49c0bf-aa2d-4f17-8a09-831d9b8bc459"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:43:48.478165 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.478114    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-kube-api-access-lc6lp" pod "7b49c0bf-aa2d-4f17-8a09-831d9b8bc459" (UID: "7b49c0bf-aa2d-4f17-8a09-831d9b8bc459"). InnerVolumeSpecName "kube-api-access-lc6lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:48.478350 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.478326    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-token" pod "7b49c0bf-aa2d-4f17-8a09-831d9b8bc459" (UID: "7b49c0bf-aa2d-4f17-8a09-831d9b8bc459"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:48.578400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.578357    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:48.578400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.578399    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc6lp\" (UniqueName: \"kubernetes.io/projected/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-kube-api-access-lc6lp\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:48.578619 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.578416    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:48.578619 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.578430    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:43:48.777952 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.777907    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fe49ab4-341a-4c8a-b993-6885741c01ae" path="/var/lib/kubelet/pods/9fe49ab4-341a-4c8a-b993-6885741c01ae/volumes"
Oct 30 22:43:48.964358 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.964323    4889 patch_prober.go:29] interesting pod/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:43:48.964530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:48.964382    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v" podUID="e2029821-7bca-434d-978b-07c37e50d52b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 22:43:49.185291 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:49.185107    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:49.295672 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:49.295590    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"
Oct 30 22:43:49.295672 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:49.295660    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk" event={"ID":"7b49c0bf-aa2d-4f17-8a09-831d9b8bc459","Type":"ContainerDied","Data":"cc8d5a3d4b48d633f0f6f2a3143bddea1dc9cc43b545f65cc3a5f355a63c03bf"}
Oct 30 22:43:49.296160 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:49.295728    4889 scope.go:122] "RemoveContainer" containerID="4429583f8ee29228a72cea60ac51ff8eafcccbf17f65a9b42882aca907b44044"
Oct 30 22:43:49.367157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:49.367121    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:49.388756 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:49.388720    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"]
Oct 30 22:43:49.393653 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:49.393618    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-2c4wk"]
Oct 30 22:43:49.964381 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:49.964347    4889 patch_prober.go:29] interesting pod/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:43:49.964556 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:49.964405    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v" podUID="e2029821-7bca-434d-978b-07c37e50d52b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 22:43:50.777565 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:50.777531    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b49c0bf-aa2d-4f17-8a09-831d9b8bc459" path="/var/lib/kubelet/pods/7b49c0bf-aa2d-4f17-8a09-831d9b8bc459/volumes"
Oct 30 22:43:50.963999 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:50.963956    4889 patch_prober.go:29] interesting pod/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:43:50.964181 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:50.964052    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v" podUID="e2029821-7bca-434d-978b-07c37e50d52b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 22:43:51.017309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:51.017267    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:51.969601 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:51.969566    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:51.970118 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:51.970093    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:51.971022 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:51.970999    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:43:52.904948 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:52.904916    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:53.200380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.200284    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"]
Oct 30 22:43:53.201214 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.201191    4889 cpu_manager.go:414] "containerMap: removing container" podUID="7b49c0bf-aa2d-4f17-8a09-831d9b8bc459" containerName="istio-proxy"
Oct 30 22:43:53.201214 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.201211    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b49c0bf-aa2d-4f17-8a09-831d9b8bc459" containerName="istio-proxy"
Oct 30 22:43:53.201214 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.201237    4889 cpu_manager.go:414] "containerMap: removing container" podUID="7b49c0bf-aa2d-4f17-8a09-831d9b8bc459" containerName="istio-proxy"
Oct 30 22:43:53.201214 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.201243    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b49c0bf-aa2d-4f17-8a09-831d9b8bc459" containerName="istio-proxy"
Oct 30 22:43:53.201214 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.201258    4889 cpu_manager.go:414] "containerMap: removing container" podUID="9fe49ab4-341a-4c8a-b993-6885741c01ae" containerName="maas-api"
Oct 30 22:43:53.201214 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.201264    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fe49ab4-341a-4c8a-b993-6885741c01ae" containerName="maas-api"
Oct 30 22:43:53.201663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.201425    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b49c0bf-aa2d-4f17-8a09-831d9b8bc459" containerName="istio-proxy"
Oct 30 22:43:53.201663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.201432    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b49c0bf-aa2d-4f17-8a09-831d9b8bc459" containerName="istio-proxy"
Oct 30 22:43:53.201663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.201443    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fe49ab4-341a-4c8a-b993-6885741c01ae" containerName="maas-api"
Oct 30 22:43:53.207036 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.207009    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"
Oct 30 22:43:53.225376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.225340    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"]
Oct 30 22:43:53.231590 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.231560    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-0ff5b0\"/\"openshift-service-ca.crt\""
Oct 30 22:43:53.238056 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.237950    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-0ff5b0\"/\"default-dockercfg-g2vql\""
Oct 30 22:43:53.238056 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.237983    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-0ff5b0\"/\"auth-test-model-0ff5b0-kserve-self-signed-certs\""
Oct 30 22:43:53.238056 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.238024    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-0ff5b0\"/\"kube-root-ca.crt\""
Oct 30 22:43:53.277209 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.277163    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21e5ac2e-104f-4c24-8e80-6fd0b457478d-tls-certs\") pod \"auth-test-model-0ff5b0-kserve-7df6875549-2nbwj\" (UID: \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\") " pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"
Oct 30 22:43:53.277408 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.277341    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-857f2\" (UniqueName: \"kubernetes.io/projected/21e5ac2e-104f-4c24-8e80-6fd0b457478d-kube-api-access-857f2\") pod \"auth-test-model-0ff5b0-kserve-7df6875549-2nbwj\" (UID: \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\") " pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"
Oct 30 22:43:53.277469 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.277435    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-model-cache\") pod \"auth-test-model-0ff5b0-kserve-7df6875549-2nbwj\" (UID: \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\") " pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"
Oct 30 22:43:53.277602 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.277584    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-tmp-dir\") pod \"auth-test-model-0ff5b0-kserve-7df6875549-2nbwj\" (UID: \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\") " pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"
Oct 30 22:43:53.277673 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.277649    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-home\") pod \"auth-test-model-0ff5b0-kserve-7df6875549-2nbwj\" (UID: \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\") " pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"
Oct 30 22:43:53.277789 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.277776    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-dshm\") pod \"auth-test-model-0ff5b0-kserve-7df6875549-2nbwj\" (UID: \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\") " pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"
Oct 30 22:43:53.324743 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.324694    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:53.379428 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.379393    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21e5ac2e-104f-4c24-8e80-6fd0b457478d-tls-certs\") pod \"auth-test-model-0ff5b0-kserve-7df6875549-2nbwj\" (UID: \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\") " pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"
Oct 30 22:43:53.379630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.379488    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-857f2\" (UniqueName: \"kubernetes.io/projected/21e5ac2e-104f-4c24-8e80-6fd0b457478d-kube-api-access-857f2\") pod \"auth-test-model-0ff5b0-kserve-7df6875549-2nbwj\" (UID: \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\") " pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"
Oct 30 22:43:53.379630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.379558    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-model-cache\") pod \"auth-test-model-0ff5b0-kserve-7df6875549-2nbwj\" (UID: \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\") " pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"
Oct 30 22:43:53.379630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.379624    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-tmp-dir\") pod \"auth-test-model-0ff5b0-kserve-7df6875549-2nbwj\" (UID: \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\") " pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"
Oct 30 22:43:53.379763 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.379665    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-home\") pod \"auth-test-model-0ff5b0-kserve-7df6875549-2nbwj\" (UID: \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\") " pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"
Oct 30 22:43:53.379815 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.379785    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-dshm\") pod \"auth-test-model-0ff5b0-kserve-7df6875549-2nbwj\" (UID: \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\") " pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"
Oct 30 22:43:53.380015 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.379991    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-model-cache\") pod \"auth-test-model-0ff5b0-kserve-7df6875549-2nbwj\" (UID: \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\") " pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"
Oct 30 22:43:53.380184 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.380034    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-tmp-dir\") pod \"auth-test-model-0ff5b0-kserve-7df6875549-2nbwj\" (UID: \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\") " pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"
Oct 30 22:43:53.380184 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.380166    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-home\") pod \"auth-test-model-0ff5b0-kserve-7df6875549-2nbwj\" (UID: \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\") " pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"
Oct 30 22:43:53.382190 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.382169    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21e5ac2e-104f-4c24-8e80-6fd0b457478d-tls-certs\") pod \"auth-test-model-0ff5b0-kserve-7df6875549-2nbwj\" (UID: \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\") " pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"
Oct 30 22:43:53.382384 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.382195    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-dshm\") pod \"auth-test-model-0ff5b0-kserve-7df6875549-2nbwj\" (UID: \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\") " pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"
Oct 30 22:43:53.402790 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.402760    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-857f2\" (UniqueName: \"kubernetes.io/projected/21e5ac2e-104f-4c24-8e80-6fd0b457478d-kube-api-access-857f2\") pod \"auth-test-model-0ff5b0-kserve-7df6875549-2nbwj\" (UID: \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\") " pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"
Oct 30 22:43:53.525138 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.525102    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"
Oct 30 22:43:53.864913 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:43:53.864881    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21e5ac2e_104f_4c24_8e80_6fd0b457478d.slice/crio-5f52f602a008eccae8a1410c2b05299193136511346fa863de36d33c7af09efb WatchSource:0}: Error finding container 5f52f602a008eccae8a1410c2b05299193136511346fa863de36d33c7af09efb: Status 404 returned error can't find the container with id 5f52f602a008eccae8a1410c2b05299193136511346fa863de36d33c7af09efb
Oct 30 22:43:53.865807 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:53.865777    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"]
Oct 30 22:43:54.347593 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:54.347549    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj" event={"ID":"21e5ac2e-104f-4c24-8e80-6fd0b457478d","Type":"ContainerStarted","Data":"3931135a1b5b9a9a39f7fc2e1515075f56049e371b0f2553291202242034c169"}
Oct 30 22:43:54.348245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:54.347609    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj" event={"ID":"21e5ac2e-104f-4c24-8e80-6fd0b457478d","Type":"ContainerStarted","Data":"5f52f602a008eccae8a1410c2b05299193136511346fa863de36d33c7af09efb"}
Oct 30 22:43:54.416381 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:54.415369    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj" podStartSLOduration=1.41534784 podStartE2EDuration="1.41534784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:54.408717667 +0000 UTC m=+2219.765083981" watchObservedRunningTime="2026-10-30 22:43:54.41534784 +0000 UTC m=+2219.771714166"
Oct 30 22:43:54.525643 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:54.525607    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"
Oct 30 22:43:54.827662 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:54.827623    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:55.005137 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:55.005099    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:55.102456 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:55.102367    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"]
Oct 30 22:43:55.102674 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:55.102644    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p" podUID="55f78a17-ccc6-4f75-902e-e048a59e6c6a" containerName="istio-proxy" containerID="cri-o://ac7e91577023d9c75d8bc84c680a58f6f90f290bdcebdc5e50ce75b934ec31f5" gracePeriod=30
Oct 30 22:43:55.192109 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:55.192075    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:56.909443 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:56.909399    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:58.714756 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:43:58.714707    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:00.402859 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.402829    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:44:00.418606 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.418575    4889 generic.go:356] "Generic (PLEG): container finished" podID="55f78a17-ccc6-4f75-902e-e048a59e6c6a" containerID="ac7e91577023d9c75d8bc84c680a58f6f90f290bdcebdc5e50ce75b934ec31f5" exitCode=0
Oct 30 22:44:00.418807 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.418625    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"
Oct 30 22:44:00.418807 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.418688    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p" event={"ID":"55f78a17-ccc6-4f75-902e-e048a59e6c6a","Type":"ContainerDied","Data":"ac7e91577023d9c75d8bc84c680a58f6f90f290bdcebdc5e50ce75b934ec31f5"}
Oct 30 22:44:00.418807 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.418758    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p" event={"ID":"55f78a17-ccc6-4f75-902e-e048a59e6c6a","Type":"ContainerDied","Data":"7edf6ec737ab614e8d39bcff3ed582c58513e738fe9e32f8b778c60d3beca356"}
Oct 30 22:44:00.418807 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.418784    4889 scope.go:122] "RemoveContainer" containerID="ac7e91577023d9c75d8bc84c680a58f6f90f290bdcebdc5e50ce75b934ec31f5"
Oct 30 22:44:00.431105 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.431084    4889 scope.go:122] "RemoveContainer" containerID="3147bbc32cb944f07e6d215f8d6c35dc3cabd7134ec9f53e91b25b94fd68bf26"
Oct 30 22:44:00.443887 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.443861    4889 scope.go:122] "RemoveContainer" containerID="ac7e91577023d9c75d8bc84c680a58f6f90f290bdcebdc5e50ce75b934ec31f5"
Oct 30 22:44:00.444280 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:44:00.444256    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac7e91577023d9c75d8bc84c680a58f6f90f290bdcebdc5e50ce75b934ec31f5\": container with ID starting with ac7e91577023d9c75d8bc84c680a58f6f90f290bdcebdc5e50ce75b934ec31f5 not found: ID does not exist" containerID="ac7e91577023d9c75d8bc84c680a58f6f90f290bdcebdc5e50ce75b934ec31f5"
Oct 30 22:44:00.444360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.444299    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac7e91577023d9c75d8bc84c680a58f6f90f290bdcebdc5e50ce75b934ec31f5"} err="failed to get container status \"ac7e91577023d9c75d8bc84c680a58f6f90f290bdcebdc5e50ce75b934ec31f5\": rpc error: code = NotFound desc = could not find container \"ac7e91577023d9c75d8bc84c680a58f6f90f290bdcebdc5e50ce75b934ec31f5\": container with ID starting with ac7e91577023d9c75d8bc84c680a58f6f90f290bdcebdc5e50ce75b934ec31f5 not found: ID does not exist"
Oct 30 22:44:00.444360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.444320    4889 scope.go:122] "RemoveContainer" containerID="3147bbc32cb944f07e6d215f8d6c35dc3cabd7134ec9f53e91b25b94fd68bf26"
Oct 30 22:44:00.444628 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:44:00.444598    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3147bbc32cb944f07e6d215f8d6c35dc3cabd7134ec9f53e91b25b94fd68bf26\": container with ID starting with 3147bbc32cb944f07e6d215f8d6c35dc3cabd7134ec9f53e91b25b94fd68bf26 not found: ID does not exist" containerID="3147bbc32cb944f07e6d215f8d6c35dc3cabd7134ec9f53e91b25b94fd68bf26"
Oct 30 22:44:00.444678 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.444636    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3147bbc32cb944f07e6d215f8d6c35dc3cabd7134ec9f53e91b25b94fd68bf26"} err="failed to get container status \"3147bbc32cb944f07e6d215f8d6c35dc3cabd7134ec9f53e91b25b94fd68bf26\": rpc error: code = NotFound desc = could not find container \"3147bbc32cb944f07e6d215f8d6c35dc3cabd7134ec9f53e91b25b94fd68bf26\": container with ID starting with 3147bbc32cb944f07e6d215f8d6c35dc3cabd7134ec9f53e91b25b94fd68bf26 not found: ID does not exist"
Oct 30 22:44:00.553388 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.553354    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f78a17-ccc6-4f75-902e-e048a59e6c6a-kube-api-access-9c68d\" (UniqueName: \"kubernetes.io/projected/55f78a17-ccc6-4f75-902e-e048a59e6c6a-kube-api-access-9c68d\") pod \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") "
Oct 30 22:44:00.553622 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.553505    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-credential-socket\") pod \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") "
Oct 30 22:44:00.553622 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.553535    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istiod-ca-cert\") pod \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") "
Oct 30 22:44:00.553622 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.553568    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-token\" (UniqueName: \"kubernetes.io/projected/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-token\") pod \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") "
Oct 30 22:44:00.553821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.553773    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-envoy\") pod \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") "
Oct 30 22:44:00.553887 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.553818    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-workload-socket\") pod \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") "
Oct 30 22:44:00.553887 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.553864    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-credential-socket" pod "55f78a17-ccc6-4f75-902e-e048a59e6c6a" (UID: "55f78a17-ccc6-4f75-902e-e048a59e6c6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:00.554012 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.553933    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-podinfo\") pod \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") "
Oct 30 22:44:00.554012 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.553955    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istiod-ca-cert" pod "55f78a17-ccc6-4f75-902e-e048a59e6c6a" (UID: "55f78a17-ccc6-4f75-902e-e048a59e6c6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:44:00.554120 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.554062    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-workload-socket" pod "55f78a17-ccc6-4f75-902e-e048a59e6c6a" (UID: "55f78a17-ccc6-4f75-902e-e048a59e6c6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:00.554575 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.554550    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-data\") pod \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") "
Oct 30 22:44:00.554775 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.554755    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-workload-certs\") pod \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\" (UID: \"55f78a17-ccc6-4f75-902e-e048a59e6c6a\") "
Oct 30 22:44:00.554997 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.554947    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-data" pod "55f78a17-ccc6-4f75-902e-e048a59e6c6a" (UID: "55f78a17-ccc6-4f75-902e-e048a59e6c6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:00.555076 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.555032    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-workload-certs" pod "55f78a17-ccc6-4f75-902e-e048a59e6c6a" (UID: "55f78a17-ccc6-4f75-902e-e048a59e6c6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:00.555700 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.555671    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:44:00.555815 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.555708    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:44:00.555815 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.555725    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:44:00.555815 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.555739    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:44:00.555815 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.555755    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:44:00.556210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.556133    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-envoy" pod "55f78a17-ccc6-4f75-902e-e048a59e6c6a" (UID: "55f78a17-ccc6-4f75-902e-e048a59e6c6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:00.556303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.556287    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f78a17-ccc6-4f75-902e-e048a59e6c6a-kube-api-access-9c68d" pod "55f78a17-ccc6-4f75-902e-e048a59e6c6a" (UID: "55f78a17-ccc6-4f75-902e-e048a59e6c6a"). InnerVolumeSpecName "kube-api-access-9c68d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:00.556380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.556363    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-podinfo" pod "55f78a17-ccc6-4f75-902e-e048a59e6c6a" (UID: "55f78a17-ccc6-4f75-902e-e048a59e6c6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:44:00.556459 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.556438    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-token" pod "55f78a17-ccc6-4f75-902e-e048a59e6c6a" (UID: "55f78a17-ccc6-4f75-902e-e048a59e6c6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:00.638010 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.637867    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:00.657851 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.657812    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c68d\" (UniqueName: \"kubernetes.io/projected/55f78a17-ccc6-4f75-902e-e048a59e6c6a-kube-api-access-9c68d\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:44:00.657851 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.657845    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:44:00.657851 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.657855    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:44:00.658189 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.657863    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f78a17-ccc6-4f75-902e-e048a59e6c6a-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:44:00.779733 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.779693    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"]
Oct 30 22:44:00.780013 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:00.779995    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-bdd94b-openshift-default-7bbf9bcd4f-q846p"]
Oct 30 22:44:02.322067 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:02.322010    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:02.777951 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:02.777915    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55f78a17-ccc6-4f75-902e-e048a59e6c6a" path="/var/lib/kubelet/pods/55f78a17-ccc6-4f75-902e-e048a59e6c6a/volumes"
Oct 30 22:44:03.526316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:03.526272    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"
Oct 30 22:44:03.542308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:03.542271    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"
Oct 30 22:44:04.012565 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:04.012514    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:04.467606 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:04.467528    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"
Oct 30 22:44:05.921383 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:05.921344    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:06.388860 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:06.388824    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:06.558334 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:06.558172    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:07.808079 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:07.808016    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:09.624287 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:09.624235    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:10.306708 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:44:10.306669    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:44:11.439610 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:11.439568    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:12.460475 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:12.460438    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:13.239967 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:13.239925    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:14.995752 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:14.994721    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:16.759042 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:16.759002    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:18.650562 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:18.650522    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:20.418480 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:20.418440    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:22.224592 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:22.224552    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:24.073968 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:24.073931    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:25.983017 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:25.982979    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:27.257800 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.257767    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"]
Oct 30 22:44:27.258959 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.258934    4889 cpu_manager.go:414] "containerMap: removing container" podUID="55f78a17-ccc6-4f75-902e-e048a59e6c6a" containerName="istio-proxy"
Oct 30 22:44:27.258959 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.258960    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f78a17-ccc6-4f75-902e-e048a59e6c6a" containerName="istio-proxy"
Oct 30 22:44:27.259145 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.259040    4889 cpu_manager.go:414] "containerMap: removing container" podUID="55f78a17-ccc6-4f75-902e-e048a59e6c6a" containerName="istio-proxy"
Oct 30 22:44:27.259145 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.259049    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f78a17-ccc6-4f75-902e-e048a59e6c6a" containerName="istio-proxy"
Oct 30 22:44:27.259300 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.259288    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f78a17-ccc6-4f75-902e-e048a59e6c6a" containerName="istio-proxy"
Oct 30 22:44:27.263837 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.263805    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"
Oct 30 22:44:27.285400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.285360    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"]
Oct 30 22:44:27.293403 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.293368    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-d9f1c2\"/\"default-dockercfg-clb4g\""
Oct 30 22:44:27.293615 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.293542    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-d9f1c2\"/\"auth-test-model-d9f1c2-kserve-self-signed-certs\""
Oct 30 22:44:27.297467 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.297439    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-d9f1c2\"/\"openshift-service-ca.crt\""
Oct 30 22:44:27.297656 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.297502    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-d9f1c2\"/\"kube-root-ca.crt\""
Oct 30 22:44:27.342995 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.342952    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-model-cache\") pod \"auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks\" (UID: \"46e8ec9f-b22e-4251-9652-118c6b52d401\") " pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"
Oct 30 22:44:27.342995 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.343004    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-home\") pod \"auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks\" (UID: \"46e8ec9f-b22e-4251-9652-118c6b52d401\") " pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"
Oct 30 22:44:27.343355 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.343242    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-dshm\") pod \"auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks\" (UID: \"46e8ec9f-b22e-4251-9652-118c6b52d401\") " pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"
Oct 30 22:44:27.343355 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.343278    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-tmp-dir\") pod \"auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks\" (UID: \"46e8ec9f-b22e-4251-9652-118c6b52d401\") " pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"
Oct 30 22:44:27.343472 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.343380    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46e8ec9f-b22e-4251-9652-118c6b52d401-tls-certs\") pod \"auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks\" (UID: \"46e8ec9f-b22e-4251-9652-118c6b52d401\") " pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"
Oct 30 22:44:27.343472 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.343399    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djvv9\" (UniqueName: \"kubernetes.io/projected/46e8ec9f-b22e-4251-9652-118c6b52d401-kube-api-access-djvv9\") pod \"auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks\" (UID: \"46e8ec9f-b22e-4251-9652-118c6b52d401\") " pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"
Oct 30 22:44:27.377627 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.377587    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:27.445575 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.445542    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-dshm\") pod \"auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks\" (UID: \"46e8ec9f-b22e-4251-9652-118c6b52d401\") " pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"
Oct 30 22:44:27.445575 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.445577    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-tmp-dir\") pod \"auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks\" (UID: \"46e8ec9f-b22e-4251-9652-118c6b52d401\") " pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"
Oct 30 22:44:27.445853 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.445602    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46e8ec9f-b22e-4251-9652-118c6b52d401-tls-certs\") pod \"auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks\" (UID: \"46e8ec9f-b22e-4251-9652-118c6b52d401\") " pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"
Oct 30 22:44:27.445853 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.445625    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djvv9\" (UniqueName: \"kubernetes.io/projected/46e8ec9f-b22e-4251-9652-118c6b52d401-kube-api-access-djvv9\") pod \"auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks\" (UID: \"46e8ec9f-b22e-4251-9652-118c6b52d401\") " pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"
Oct 30 22:44:27.445996 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.445974    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-model-cache\") pod \"auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks\" (UID: \"46e8ec9f-b22e-4251-9652-118c6b52d401\") " pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"
Oct 30 22:44:27.446068 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.446025    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-home\") pod \"auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks\" (UID: \"46e8ec9f-b22e-4251-9652-118c6b52d401\") " pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"
Oct 30 22:44:27.446134 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.446117    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-tmp-dir\") pod \"auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks\" (UID: \"46e8ec9f-b22e-4251-9652-118c6b52d401\") " pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"
Oct 30 22:44:27.446371 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.446337    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-model-cache\") pod \"auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks\" (UID: \"46e8ec9f-b22e-4251-9652-118c6b52d401\") " pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"
Oct 30 22:44:27.446674 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.446404    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-home\") pod \"auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks\" (UID: \"46e8ec9f-b22e-4251-9652-118c6b52d401\") " pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"
Oct 30 22:44:27.448345 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.448315    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-dshm\") pod \"auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks\" (UID: \"46e8ec9f-b22e-4251-9652-118c6b52d401\") " pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"
Oct 30 22:44:27.448517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.448498    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46e8ec9f-b22e-4251-9652-118c6b52d401-tls-certs\") pod \"auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks\" (UID: \"46e8ec9f-b22e-4251-9652-118c6b52d401\") " pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"
Oct 30 22:44:27.468399 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.468364    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djvv9\" (UniqueName: \"kubernetes.io/projected/46e8ec9f-b22e-4251-9652-118c6b52d401-kube-api-access-djvv9\") pod \"auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks\" (UID: \"46e8ec9f-b22e-4251-9652-118c6b52d401\") " pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"
Oct 30 22:44:27.582639 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.582464    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"
Oct 30 22:44:27.906750 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:27.906719    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"]
Oct 30 22:44:27.907901 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:44:27.907874    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46e8ec9f_b22e_4251_9652_118c6b52d401.slice/crio-1f87aa62e4617af5ea1b9ded9f8d3dd8436d78305bc5ca0ae4091826fd4f301b WatchSource:0}: Error finding container 1f87aa62e4617af5ea1b9ded9f8d3dd8436d78305bc5ca0ae4091826fd4f301b: Status 404 returned error can't find the container with id 1f87aa62e4617af5ea1b9ded9f8d3dd8436d78305bc5ca0ae4091826fd4f301b
Oct 30 22:44:28.062499 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:28.062458    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:28.698362 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:28.698318    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks" event={"ID":"46e8ec9f-b22e-4251-9652-118c6b52d401","Type":"ContainerStarted","Data":"f63015797021c1648ab18d9eb2dccd7aa32abfd7dacb318ceb380da7f1408262"}
Oct 30 22:44:28.698362 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:28.698376    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks" event={"ID":"46e8ec9f-b22e-4251-9652-118c6b52d401","Type":"ContainerStarted","Data":"1f87aa62e4617af5ea1b9ded9f8d3dd8436d78305bc5ca0ae4091826fd4f301b"}
Oct 30 22:44:28.765931 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:28.765865    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks" podStartSLOduration=1.765848864 podStartE2EDuration="1.765848864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:28.760530863 +0000 UTC m=+2254.116897180" watchObservedRunningTime="2026-10-30 22:44:28.765848864 +0000 UTC m=+2254.122215179"
Oct 30 22:44:29.583610 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:29.583574    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"
Oct 30 22:44:29.853516 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:29.853425    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:31.683656 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:31.683618    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:33.519597 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:33.519558    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:35.190465 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:35.190426    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:36.830691 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:36.830654    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:37.582917 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:37.582876    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"
Oct 30 22:44:37.597866 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:37.597834    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"
Oct 30 22:44:37.789508 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:37.789475    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"
Oct 30 22:44:38.734262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:38.734204    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:40.304854 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:44:40.304811    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:44:40.488420 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:40.488332    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:42.229976 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:42.229942    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:43.946604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:43.946550    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:45.716465 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:45.716422    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:47.113133 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:47.113091    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h"]
Oct 30 22:44:47.113718 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:47.113375    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h" podUID="d2241313-b99d-4804-9c2e-eb54319c1668" containerName="maas-api" containerID="cri-o://945b7df290f39c5bc7474c4b7dd023593f26ef821a30ae091da6f6d62d07f8b5" gracePeriod=30
Oct 30 22:44:47.285629 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:47.285591    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:47.503704 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:47.503614    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h"
Oct 30 22:44:47.607435 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:47.607397    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2241313-b99d-4804-9c2e-eb54319c1668-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2241313-b99d-4804-9c2e-eb54319c1668-metrics-certs\") pod \"d2241313-b99d-4804-9c2e-eb54319c1668\" (UID: \"d2241313-b99d-4804-9c2e-eb54319c1668\") "
Oct 30 22:44:47.607897 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:47.607872    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2241313-b99d-4804-9c2e-eb54319c1668-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2241313-b99d-4804-9c2e-eb54319c1668-openshift-service-ca\") pod \"d2241313-b99d-4804-9c2e-eb54319c1668\" (UID: \"d2241313-b99d-4804-9c2e-eb54319c1668\") "
Oct 30 22:44:47.608056 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:47.608042    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2241313-b99d-4804-9c2e-eb54319c1668-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2241313-b99d-4804-9c2e-eb54319c1668-maas-api-tls\") pod \"d2241313-b99d-4804-9c2e-eb54319c1668\" (UID: \"d2241313-b99d-4804-9c2e-eb54319c1668\") "
Oct 30 22:44:47.608378 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:47.608357    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2241313-b99d-4804-9c2e-eb54319c1668-kube-api-access-vls9f\" (UniqueName: \"kubernetes.io/projected/d2241313-b99d-4804-9c2e-eb54319c1668-kube-api-access-vls9f\") pod \"d2241313-b99d-4804-9c2e-eb54319c1668\" (UID: \"d2241313-b99d-4804-9c2e-eb54319c1668\") "
Oct 30 22:44:47.612925 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:47.612893    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2241313-b99d-4804-9c2e-eb54319c1668-openshift-service-ca" pod "d2241313-b99d-4804-9c2e-eb54319c1668" (UID: "d2241313-b99d-4804-9c2e-eb54319c1668"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:44:47.633469 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:47.633395    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2241313-b99d-4804-9c2e-eb54319c1668-metrics-certs" pod "d2241313-b99d-4804-9c2e-eb54319c1668" (UID: "d2241313-b99d-4804-9c2e-eb54319c1668"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:44:47.633469 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:47.633409    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2241313-b99d-4804-9c2e-eb54319c1668-maas-api-tls" pod "d2241313-b99d-4804-9c2e-eb54319c1668" (UID: "d2241313-b99d-4804-9c2e-eb54319c1668"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:44:47.633469 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:47.633451    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2241313-b99d-4804-9c2e-eb54319c1668-kube-api-access-vls9f" pod "d2241313-b99d-4804-9c2e-eb54319c1668" (UID: "d2241313-b99d-4804-9c2e-eb54319c1668"). InnerVolumeSpecName "kube-api-access-vls9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:47.711440 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:47.711397    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2241313-b99d-4804-9c2e-eb54319c1668-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:44:47.711440 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:47.711438    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2241313-b99d-4804-9c2e-eb54319c1668-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:44:47.711440 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:47.711451    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2241313-b99d-4804-9c2e-eb54319c1668-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:44:47.711832 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:47.711463    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vls9f\" (UniqueName: \"kubernetes.io/projected/d2241313-b99d-4804-9c2e-eb54319c1668-kube-api-access-vls9f\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:44:47.863492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:47.863394    4889 generic.go:356] "Generic (PLEG): container finished" podID="d2241313-b99d-4804-9c2e-eb54319c1668" containerID="945b7df290f39c5bc7474c4b7dd023593f26ef821a30ae091da6f6d62d07f8b5" exitCode=0
Oct 30 22:44:47.863492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:47.863453    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h"
Oct 30 22:44:47.863492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:47.863478    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h" event={"ID":"d2241313-b99d-4804-9c2e-eb54319c1668","Type":"ContainerDied","Data":"945b7df290f39c5bc7474c4b7dd023593f26ef821a30ae091da6f6d62d07f8b5"}
Oct 30 22:44:47.863789 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:47.863538    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h" event={"ID":"d2241313-b99d-4804-9c2e-eb54319c1668","Type":"ContainerDied","Data":"289706a67152eedbbb479b0037f114fbc6d0a51444ec3333d9a1dc7c8dd2c8c1"}
Oct 30 22:44:47.863789 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:47.863560    4889 scope.go:122] "RemoveContainer" containerID="945b7df290f39c5bc7474c4b7dd023593f26ef821a30ae091da6f6d62d07f8b5"
Oct 30 22:44:47.877512 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:47.877479    4889 scope.go:122] "RemoveContainer" containerID="945b7df290f39c5bc7474c4b7dd023593f26ef821a30ae091da6f6d62d07f8b5"
Oct 30 22:44:47.877898 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:44:47.877872    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"945b7df290f39c5bc7474c4b7dd023593f26ef821a30ae091da6f6d62d07f8b5\": container with ID starting with 945b7df290f39c5bc7474c4b7dd023593f26ef821a30ae091da6f6d62d07f8b5 not found: ID does not exist" containerID="945b7df290f39c5bc7474c4b7dd023593f26ef821a30ae091da6f6d62d07f8b5"
Oct 30 22:44:47.877959 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:47.877906    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945b7df290f39c5bc7474c4b7dd023593f26ef821a30ae091da6f6d62d07f8b5"} err="failed to get container status \"945b7df290f39c5bc7474c4b7dd023593f26ef821a30ae091da6f6d62d07f8b5\": rpc error: code = NotFound desc = could not find container \"945b7df290f39c5bc7474c4b7dd023593f26ef821a30ae091da6f6d62d07f8b5\": container with ID starting with 945b7df290f39c5bc7474c4b7dd023593f26ef821a30ae091da6f6d62d07f8b5 not found: ID does not exist"
Oct 30 22:44:47.903539 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:47.903503    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:47.925364 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:47.925328    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h"]
Oct 30 22:44:47.933958 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:47.933926    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6013c8-585976c6db-hfk4h"]
Oct 30 22:44:48.079033 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:48.078994    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:48.535597 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:48.535560    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:48.777467 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:48.777429    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2241313-b99d-4804-9c2e-eb54319c1668" path="/var/lib/kubelet/pods/d2241313-b99d-4804-9c2e-eb54319c1668/volumes"
Oct 30 22:44:49.489758 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:49.489719    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:49.998649 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:49.998614    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:51.742206 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:51.742167    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:53.521741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:53.521694    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:54.464468 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:54.464433    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:54.841669 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:54.841610    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:55.031133 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:55.031068    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:55.200908 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:55.200779    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:55.374319 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:55.369061    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:56.007247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.007196    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:56.572525 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.572476    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"]
Oct 30 22:44:56.581890 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.574335    4889 cpu_manager.go:414] "containerMap: removing container" podUID="d2241313-b99d-4804-9c2e-eb54319c1668" containerName="maas-api"
Oct 30 22:44:56.581890 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.575404    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2241313-b99d-4804-9c2e-eb54319c1668" containerName="maas-api"
Oct 30 22:44:56.581890 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.575626    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2241313-b99d-4804-9c2e-eb54319c1668" containerName="maas-api"
Oct 30 22:44:56.581890 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.575637    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f78a17-ccc6-4f75-902e-e048a59e6c6a" containerName="istio-proxy"
Oct 30 22:44:56.585036 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.584961    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"]
Oct 30 22:44:56.585273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.585088    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.603419 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.603385    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-c511a3-openshift-default-dockercfg-j6vc7\""
Oct 30 22:44:56.654850 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.654793    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:56.753194 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.752087    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-workload-socket\") pod \"e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") " pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.753194 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.752321    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-credential-socket\") pod \"e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") " pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.753194 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.752366    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-istio-envoy\") pod \"e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") " pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.753194 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.752401    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-workload-certs\") pod \"e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") " pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.753194 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.752429    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a707459-e53d-486b-a1e5-0456ab08570c-istio-podinfo\") pod \"e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") " pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.753194 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.752453    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a707459-e53d-486b-a1e5-0456ab08570c-istio-token\") pod \"e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") " pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.753194 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.752620    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a707459-e53d-486b-a1e5-0456ab08570c-istiod-ca-cert\") pod \"e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") " pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.753194 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.752794    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-istio-data\") pod \"e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") " pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.753194 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.752859    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqmtc\" (UniqueName: \"kubernetes.io/projected/5a707459-e53d-486b-a1e5-0456ab08570c-kube-api-access-xqmtc\") pod \"e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") " pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.855568 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.855432    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-workload-socket\") pod \"e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") " pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.855778 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.855695    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-credential-socket\") pod \"e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") " pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.855778 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.855754    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-istio-envoy\") pod \"e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") " pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.855932 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.855789    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-workload-certs\") pod \"e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") " pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.855932 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.855820    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a707459-e53d-486b-a1e5-0456ab08570c-istio-podinfo\") pod \"e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") " pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.855932 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.855848    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a707459-e53d-486b-a1e5-0456ab08570c-istio-token\") pod \"e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") " pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.855932 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.855903    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-workload-socket\") pod \"e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") " pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.856151 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.856069    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a707459-e53d-486b-a1e5-0456ab08570c-istiod-ca-cert\") pod \"e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") " pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.856371 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.856204    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-istio-data\") pod \"e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") " pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.856371 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.856336    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqmtc\" (UniqueName: \"kubernetes.io/projected/5a707459-e53d-486b-a1e5-0456ab08570c-kube-api-access-xqmtc\") pod \"e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") " pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.856547 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.856381    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-workload-certs\") pod \"e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") " pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.856961 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.856941    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-credential-socket\") pod \"e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") " pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.857159 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.857138    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-istio-data\") pod \"e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") " pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.858046 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.858017    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a707459-e53d-486b-a1e5-0456ab08570c-istiod-ca-cert\") pod \"e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") " pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.859067 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.859038    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a707459-e53d-486b-a1e5-0456ab08570c-istio-podinfo\") pod \"e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") " pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.860309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.860290    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-istio-envoy\") pod \"e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") " pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.878305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.878267    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a707459-e53d-486b-a1e5-0456ab08570c-istio-token\") pod \"e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") " pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.878459 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.878367    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqmtc\" (UniqueName: \"kubernetes.io/projected/5a707459-e53d-486b-a1e5-0456ab08570c-kube-api-access-xqmtc\") pod \"e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") " pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:56.893427 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.893381    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:56.907447 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:56.907412    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:57.229302 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:57.229155    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:57.247519 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:57.247492    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"]
Oct 30 22:44:57.249750 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:44:57.249720    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a707459_e53d_486b_a1e5_0456ab08570c.slice/crio-65714b706e2728e74b35ec51b428bf1594b8bdc800406551ef8e76c706665cf5 WatchSource:0}: Error finding container 65714b706e2728e74b35ec51b428bf1594b8bdc800406551ef8e76c706665cf5: Status 404 returned error can't find the container with id 65714b706e2728e74b35ec51b428bf1594b8bdc800406551ef8e76c706665cf5
Oct 30 22:44:57.253271 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:57.253175    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:44:57.253406 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:57.253308    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:44:57.253406 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:57.253366    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:44:57.953908 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:57.953869    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr" event={"ID":"5a707459-e53d-486b-a1e5-0456ab08570c","Type":"ContainerStarted","Data":"981bd4e0cb5bd05d6b76026ccab69712b1d990a628fddfae3a194e1f3c7d624c"}
Oct 30 22:44:57.954103 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:57.953923    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr" event={"ID":"5a707459-e53d-486b-a1e5-0456ab08570c","Type":"ContainerStarted","Data":"65714b706e2728e74b35ec51b428bf1594b8bdc800406551ef8e76c706665cf5"}
Oct 30 22:44:58.030479 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:58.030416    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr" podStartSLOduration=2.03039705 podStartE2EDuration="2.03039705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:58.012898684 +0000 UTC m=+2283.369265028" watchObservedRunningTime="2026-10-30 22:44:58.03039705 +0000 UTC m=+2283.386763364"
Oct 30 22:44:58.908310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:58.908214    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:58.915368 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:58.915332    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:58.966809 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:58.966775    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:58.967877 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:58.967854    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:44:59.045780 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:44:59.045744    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:00.201560 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.201523    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8"]
Oct 30 22:45:00.208588 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.208554    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8"
Oct 30 22:45:00.280315 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.280280    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8"]
Oct 30 22:45:00.315990 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.315953    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:45:00.321692 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.321655    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g"]
Oct 30 22:45:00.329376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.329342    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g"
Oct 30 22:45:00.331466 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.331328    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw"]
Oct 30 22:45:00.340618 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.340541    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-ww2qg"]
Oct 30 22:45:00.343199 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.341983    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw"
Oct 30 22:45:00.348927 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.345850    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d07912-2775-4414-9754-d3e16bdf69d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8\" (UID: \"23d07912-2775-4414-9754-d3e16bdf69d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8"
Oct 30 22:45:00.348927 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.345936    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9527p\" (UniqueName: \"kubernetes.io/projected/23d07912-2775-4414-9754-d3e16bdf69d2-kube-api-access-9527p\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8\" (UID: \"23d07912-2775-4414-9754-d3e16bdf69d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8"
Oct 30 22:45:00.352306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.352128    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g"]
Oct 30 22:45:00.352306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.352177    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f"]
Oct 30 22:45:00.354413 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.354382    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-ww2qg"
Oct 30 22:45:00.362848 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.362822    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-ww2qg"]
Oct 30 22:45:00.363009 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.362863    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d"]
Oct 30 22:45:00.363721 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.363635    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f"
Oct 30 22:45:00.368811 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.368789    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d"
Oct 30 22:45:00.371031 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.371009    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr"]
Oct 30 22:45:00.376453 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.376427    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl"]
Oct 30 22:45:00.376825 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.376805    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr"
Oct 30 22:45:00.381490 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.381468    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl"
Oct 30 22:45:00.386588 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.386564    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f"]
Oct 30 22:45:00.392303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.392269    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d"]
Oct 30 22:45:00.395502 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.395468    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj"]
Oct 30 22:45:00.402437 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.402109    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw"]
Oct 30 22:45:00.402437 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.402247    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj"
Oct 30 22:45:00.407772 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.407743    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl"]
Oct 30 22:45:00.414552 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.414269    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj"]
Oct 30 22:45:00.418896 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.418869    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr"]
Oct 30 22:45:00.448683 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.448633    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b83f41b5-12fa-4cd6-896c-5639ead76511-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f\" (UID: \"b83f41b5-12fa-4cd6-896c-5639ead76511\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f"
Oct 30 22:45:00.448884 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.448705    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrsn6\" (UniqueName: \"kubernetes.io/projected/78b13226-d659-4865-9627-024fd2ced764-kube-api-access-wrsn6\") pod \"maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl\" (UID: \"78b13226-d659-4865-9627-024fd2ced764\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl"
Oct 30 22:45:00.448884 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.448859    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2csbg\" (UniqueName: \"kubernetes.io/projected/ec7b92fc-0bac-45c3-9adb-b0cd06b33198-kube-api-access-2csbg\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr\" (UID: \"ec7b92fc-0bac-45c3-9adb-b0cd06b33198\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr"
Oct 30 22:45:00.449019 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.448928    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqjts\" (UniqueName: \"kubernetes.io/projected/62aab7d6-3e77-4389-8f85-ce60b5f90056-kube-api-access-tqjts\") pod \"maas-api-key-cleanup-29890005-ww2qg\" (UID: \"62aab7d6-3e77-4389-8f85-ce60b5f90056\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-ww2qg"
Oct 30 22:45:00.449090 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.449030    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d953f06-b11d-4115-8b64-3970e1639126-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw\" (UID: \"3d953f06-b11d-4115-8b64-3970e1639126\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw"
Oct 30 22:45:00.449205 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.449096    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3530f8ff-0e20-4062-8939-aa465f065f24-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d\" (UID: \"3530f8ff-0e20-4062-8939-aa465f065f24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d"
Oct 30 22:45:00.449205 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.449188    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7b92fc-0bac-45c3-9adb-b0cd06b33198-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr\" (UID: \"ec7b92fc-0bac-45c3-9adb-b0cd06b33198\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr"
Oct 30 22:45:00.449553 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.449271    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q49gh\" (UniqueName: \"kubernetes.io/projected/dfe9d27e-3639-4085-9f3d-93d8a9ec5063-kube-api-access-q49gh\") pod \"maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj\" (UID: \"dfe9d27e-3639-4085-9f3d-93d8a9ec5063\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj"
Oct 30 22:45:00.449553 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.449357    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g\" (UID: \"50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g"
Oct 30 22:45:00.449553 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.449382    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtj4r\" (UniqueName: \"kubernetes.io/projected/3d953f06-b11d-4115-8b64-3970e1639126-kube-api-access-rtj4r\") pod \"maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw\" (UID: \"3d953f06-b11d-4115-8b64-3970e1639126\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw"
Oct 30 22:45:00.449553 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.449421    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88t7r\" (UniqueName: \"kubernetes.io/projected/3530f8ff-0e20-4062-8939-aa465f065f24-kube-api-access-88t7r\") pod \"maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d\" (UID: \"3530f8ff-0e20-4062-8939-aa465f065f24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d"
Oct 30 22:45:00.449553 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.449485    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe9d27e-3639-4085-9f3d-93d8a9ec5063-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj\" (UID: \"dfe9d27e-3639-4085-9f3d-93d8a9ec5063\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj"
Oct 30 22:45:00.449553 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.449521    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwbkp\" (UniqueName: \"kubernetes.io/projected/b83f41b5-12fa-4cd6-896c-5639ead76511-kube-api-access-kwbkp\") pod \"maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f\" (UID: \"b83f41b5-12fa-4cd6-896c-5639ead76511\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f"
Oct 30 22:45:00.449970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.449756    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d07912-2775-4414-9754-d3e16bdf69d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8\" (UID: \"23d07912-2775-4414-9754-d3e16bdf69d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8"
Oct 30 22:45:00.449970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.449798    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78b13226-d659-4865-9627-024fd2ced764-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl\" (UID: \"78b13226-d659-4865-9627-024fd2ced764\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl"
Oct 30 22:45:00.449970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.449877    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9527p\" (UniqueName: \"kubernetes.io/projected/23d07912-2775-4414-9754-d3e16bdf69d2-kube-api-access-9527p\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8\" (UID: \"23d07912-2775-4414-9754-d3e16bdf69d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8"
Oct 30 22:45:00.449970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.449927    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62aab7d6-3e77-4389-8f85-ce60b5f90056-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-ww2qg\" (UID: \"62aab7d6-3e77-4389-8f85-ce60b5f90056\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-ww2qg"
Oct 30 22:45:00.449970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.449952    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cw7q\" (UniqueName: \"kubernetes.io/projected/50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa-kube-api-access-8cw7q\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g\" (UID: \"50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g"
Oct 30 22:45:00.450763 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.450740    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d07912-2775-4414-9754-d3e16bdf69d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8\" (UID: \"23d07912-2775-4414-9754-d3e16bdf69d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8"
Oct 30 22:45:00.472732 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.472639    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9527p\" (UniqueName: \"kubernetes.io/projected/23d07912-2775-4414-9754-d3e16bdf69d2-kube-api-access-9527p\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8\" (UID: \"23d07912-2775-4414-9754-d3e16bdf69d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8"
Oct 30 22:45:00.486297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.486211    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:00.551583 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.551544    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62aab7d6-3e77-4389-8f85-ce60b5f90056-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-ww2qg\" (UID: \"62aab7d6-3e77-4389-8f85-ce60b5f90056\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-ww2qg"
Oct 30 22:45:00.551583 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.551583    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cw7q\" (UniqueName: \"kubernetes.io/projected/50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa-kube-api-access-8cw7q\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g\" (UID: \"50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g"
Oct 30 22:45:00.551835 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.551643    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b83f41b5-12fa-4cd6-896c-5639ead76511-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f\" (UID: \"b83f41b5-12fa-4cd6-896c-5639ead76511\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f"
Oct 30 22:45:00.551835 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.551676    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrsn6\" (UniqueName: \"kubernetes.io/projected/78b13226-d659-4865-9627-024fd2ced764-kube-api-access-wrsn6\") pod \"maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl\" (UID: \"78b13226-d659-4865-9627-024fd2ced764\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl"
Oct 30 22:45:00.551835 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.551759    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2csbg\" (UniqueName: \"kubernetes.io/projected/ec7b92fc-0bac-45c3-9adb-b0cd06b33198-kube-api-access-2csbg\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr\" (UID: \"ec7b92fc-0bac-45c3-9adb-b0cd06b33198\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr"
Oct 30 22:45:00.552025 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.551853    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqjts\" (UniqueName: \"kubernetes.io/projected/62aab7d6-3e77-4389-8f85-ce60b5f90056-kube-api-access-tqjts\") pod \"maas-api-key-cleanup-29890005-ww2qg\" (UID: \"62aab7d6-3e77-4389-8f85-ce60b5f90056\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-ww2qg"
Oct 30 22:45:00.552025 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.551891    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d953f06-b11d-4115-8b64-3970e1639126-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw\" (UID: \"3d953f06-b11d-4115-8b64-3970e1639126\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw"
Oct 30 22:45:00.552025 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.551942    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3530f8ff-0e20-4062-8939-aa465f065f24-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d\" (UID: \"3530f8ff-0e20-4062-8939-aa465f065f24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d"
Oct 30 22:45:00.552025 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.551973    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7b92fc-0bac-45c3-9adb-b0cd06b33198-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr\" (UID: \"ec7b92fc-0bac-45c3-9adb-b0cd06b33198\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr"
Oct 30 22:45:00.552025 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.552015    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q49gh\" (UniqueName: \"kubernetes.io/projected/dfe9d27e-3639-4085-9f3d-93d8a9ec5063-kube-api-access-q49gh\") pod \"maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj\" (UID: \"dfe9d27e-3639-4085-9f3d-93d8a9ec5063\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj"
Oct 30 22:45:00.552413 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.552095    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g\" (UID: \"50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g"
Oct 30 22:45:00.552413 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.552121    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtj4r\" (UniqueName: \"kubernetes.io/projected/3d953f06-b11d-4115-8b64-3970e1639126-kube-api-access-rtj4r\") pod \"maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw\" (UID: \"3d953f06-b11d-4115-8b64-3970e1639126\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw"
Oct 30 22:45:00.552413 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.552155    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88t7r\" (UniqueName: \"kubernetes.io/projected/3530f8ff-0e20-4062-8939-aa465f065f24-kube-api-access-88t7r\") pod \"maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d\" (UID: \"3530f8ff-0e20-4062-8939-aa465f065f24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d"
Oct 30 22:45:00.552413 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.552209    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe9d27e-3639-4085-9f3d-93d8a9ec5063-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj\" (UID: \"dfe9d27e-3639-4085-9f3d-93d8a9ec5063\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj"
Oct 30 22:45:00.552413 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.552348    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwbkp\" (UniqueName: \"kubernetes.io/projected/b83f41b5-12fa-4cd6-896c-5639ead76511-kube-api-access-kwbkp\") pod \"maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f\" (UID: \"b83f41b5-12fa-4cd6-896c-5639ead76511\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f"
Oct 30 22:45:00.552413 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.552387    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78b13226-d659-4865-9627-024fd2ced764-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl\" (UID: \"78b13226-d659-4865-9627-024fd2ced764\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl"
Oct 30 22:45:00.552766 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.552556    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b83f41b5-12fa-4cd6-896c-5639ead76511-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f\" (UID: \"b83f41b5-12fa-4cd6-896c-5639ead76511\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f"
Oct 30 22:45:00.552766 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.552566    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62aab7d6-3e77-4389-8f85-ce60b5f90056-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-ww2qg\" (UID: \"62aab7d6-3e77-4389-8f85-ce60b5f90056\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-ww2qg"
Oct 30 22:45:00.552766 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.552702    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d953f06-b11d-4115-8b64-3970e1639126-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw\" (UID: \"3d953f06-b11d-4115-8b64-3970e1639126\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw"
Oct 30 22:45:00.553193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.552807    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7b92fc-0bac-45c3-9adb-b0cd06b33198-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr\" (UID: \"ec7b92fc-0bac-45c3-9adb-b0cd06b33198\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr"
Oct 30 22:45:00.553193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.552972    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g\" (UID: \"50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g"
Oct 30 22:45:00.553193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.553178    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78b13226-d659-4865-9627-024fd2ced764-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl\" (UID: \"78b13226-d659-4865-9627-024fd2ced764\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl"
Oct 30 22:45:00.553422 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.553324    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3530f8ff-0e20-4062-8939-aa465f065f24-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d\" (UID: \"3530f8ff-0e20-4062-8939-aa465f065f24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d"
Oct 30 22:45:00.553422 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.553354    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe9d27e-3639-4085-9f3d-93d8a9ec5063-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj\" (UID: \"dfe9d27e-3639-4085-9f3d-93d8a9ec5063\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj"
Oct 30 22:45:00.577088 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.577050    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2csbg\" (UniqueName: \"kubernetes.io/projected/ec7b92fc-0bac-45c3-9adb-b0cd06b33198-kube-api-access-2csbg\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr\" (UID: \"ec7b92fc-0bac-45c3-9adb-b0cd06b33198\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr"
Oct 30 22:45:00.577358 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.577125    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88t7r\" (UniqueName: \"kubernetes.io/projected/3530f8ff-0e20-4062-8939-aa465f065f24-kube-api-access-88t7r\") pod \"maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d\" (UID: \"3530f8ff-0e20-4062-8939-aa465f065f24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d"
Oct 30 22:45:00.577584 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.577556    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwbkp\" (UniqueName: \"kubernetes.io/projected/b83f41b5-12fa-4cd6-896c-5639ead76511-kube-api-access-kwbkp\") pod \"maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f\" (UID: \"b83f41b5-12fa-4cd6-896c-5639ead76511\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f"
Oct 30 22:45:00.578166 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.578141    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtj4r\" (UniqueName: \"kubernetes.io/projected/3d953f06-b11d-4115-8b64-3970e1639126-kube-api-access-rtj4r\") pod \"maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw\" (UID: \"3d953f06-b11d-4115-8b64-3970e1639126\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw"
Oct 30 22:45:00.578362 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.578332    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cw7q\" (UniqueName: \"kubernetes.io/projected/50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa-kube-api-access-8cw7q\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g\" (UID: \"50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g"
Oct 30 22:45:00.578670 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.578642    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrsn6\" (UniqueName: \"kubernetes.io/projected/78b13226-d659-4865-9627-024fd2ced764-kube-api-access-wrsn6\") pod \"maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl\" (UID: \"78b13226-d659-4865-9627-024fd2ced764\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl"
Oct 30 22:45:00.578770 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.578747    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqjts\" (UniqueName: \"kubernetes.io/projected/62aab7d6-3e77-4389-8f85-ce60b5f90056-kube-api-access-tqjts\") pod \"maas-api-key-cleanup-29890005-ww2qg\" (UID: \"62aab7d6-3e77-4389-8f85-ce60b5f90056\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-ww2qg"
Oct 30 22:45:00.578856 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.578831    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q49gh\" (UniqueName: \"kubernetes.io/projected/dfe9d27e-3639-4085-9f3d-93d8a9ec5063-kube-api-access-q49gh\") pod \"maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj\" (UID: \"dfe9d27e-3639-4085-9f3d-93d8a9ec5063\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj"
Oct 30 22:45:00.580636 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.580607    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8"
Oct 30 22:45:00.661874 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.661384    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g"
Oct 30 22:45:00.663642 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.663592    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:00.671057 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.670816    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw"
Oct 30 22:45:00.694915 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.694646    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-ww2qg"
Oct 30 22:45:00.713323 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.713212    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f"
Oct 30 22:45:00.725892 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.725636    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d"
Oct 30 22:45:00.735063 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.734790    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr"
Oct 30 22:45:00.747114 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.746655    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl"
Oct 30 22:45:00.759306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.758947    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj"
Oct 30 22:45:00.911908 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:00.911853    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8"]
Oct 30 22:45:01.006368 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:01.006288    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8" event={"ID":"23d07912-2775-4414-9754-d3e16bdf69d2","Type":"ContainerStarted","Data":"859b34ebd536dec298e03ccc442edc11c86b257bdc9d3c35b70e5ee0885ea9c4"}
Oct 30 22:45:01.158206 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:01.158169    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:01.240731 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:45:01.240698    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ccda0b_60a9_4d7c_a265_9fdeb90bcdaa.slice/crio-5b50ac9ce178e64ca4b1d16f45edcb8b817a8def87010e4a1d66ad34318ba22d WatchSource:0}: Error finding container 5b50ac9ce178e64ca4b1d16f45edcb8b817a8def87010e4a1d66ad34318ba22d: Status 404 returned error can't find the container with id 5b50ac9ce178e64ca4b1d16f45edcb8b817a8def87010e4a1d66ad34318ba22d
Oct 30 22:45:01.244370 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:01.244343    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g"]
Oct 30 22:45:01.249710 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:01.249680    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw"]
Oct 30 22:45:01.250045 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:45:01.250025    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d953f06_b11d_4115_8b64_3970e1639126.slice/crio-225d75b05c61206c97e73a87ca4f220f723340cf25efdfc542aa877db4e3b142 WatchSource:0}: Error finding container 225d75b05c61206c97e73a87ca4f220f723340cf25efdfc542aa877db4e3b142: Status 404 returned error can't find the container with id 225d75b05c61206c97e73a87ca4f220f723340cf25efdfc542aa877db4e3b142
Oct 30 22:45:01.325034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:01.324995    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-ww2qg"]
Oct 30 22:45:01.341680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:01.341644    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d"]
Oct 30 22:45:01.348447 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:01.348413    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f"]
Oct 30 22:45:01.399360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:01.399323    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr"]
Oct 30 22:45:01.420821 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:45:01.420784    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb83f41b5_12fa_4cd6_896c_5639ead76511.slice/crio-593c13867d0a170accb6f2eed7864cb10f8d456651628edd137dff3cb1443d2a WatchSource:0}: Error finding container 593c13867d0a170accb6f2eed7864cb10f8d456651628edd137dff3cb1443d2a: Status 404 returned error can't find the container with id 593c13867d0a170accb6f2eed7864cb10f8d456651628edd137dff3cb1443d2a
Oct 30 22:45:01.422572 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:45:01.422544    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3530f8ff_0e20_4062_8939_aa465f065f24.slice/crio-1214b95bae1340ac830544cec060f8d0a1a8b83b94e5d1f1e8a59275e4246106 WatchSource:0}: Error finding container 1214b95bae1340ac830544cec060f8d0a1a8b83b94e5d1f1e8a59275e4246106: Status 404 returned error can't find the container with id 1214b95bae1340ac830544cec060f8d0a1a8b83b94e5d1f1e8a59275e4246106
Oct 30 22:45:01.423941 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:01.423908    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj"]
Oct 30 22:45:01.424762 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:45:01.424724    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec7b92fc_0bac_45c3_9adb_b0cd06b33198.slice/crio-666c13fdb6b72dfa6d17572f0b64c7523fc7395309ed6c57b13fcc5ed1804fe4 WatchSource:0}: Error finding container 666c13fdb6b72dfa6d17572f0b64c7523fc7395309ed6c57b13fcc5ed1804fe4: Status 404 returned error can't find the container with id 666c13fdb6b72dfa6d17572f0b64c7523fc7395309ed6c57b13fcc5ed1804fe4
Oct 30 22:45:01.429014 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:45:01.426668    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe9d27e_3639_4085_9f3d_93d8a9ec5063.slice/crio-94b55c841fcf7e59788cbdbc53c96d71179b5f370b0c0b4e9fe7b5e00320a04e WatchSource:0}: Error finding container 94b55c841fcf7e59788cbdbc53c96d71179b5f370b0c0b4e9fe7b5e00320a04e: Status 404 returned error can't find the container with id 94b55c841fcf7e59788cbdbc53c96d71179b5f370b0c0b4e9fe7b5e00320a04e
Oct 30 22:45:01.433109 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:01.433074    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl"]
Oct 30 22:45:01.440895 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:45:01.440848    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b13226_d659_4865_9627_024fd2ced764.slice/crio-be10524d95672d8b31ef45b37b26a3a31ae40ee998d496ace0794cb8e4500cb0 WatchSource:0}: Error finding container be10524d95672d8b31ef45b37b26a3a31ae40ee998d496ace0794cb8e4500cb0: Status 404 returned error can't find the container with id be10524d95672d8b31ef45b37b26a3a31ae40ee998d496ace0794cb8e4500cb0
Oct 30 22:45:02.019319 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.019279    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8" event={"ID":"23d07912-2775-4414-9754-d3e16bdf69d2","Type":"ContainerStarted","Data":"54fd5e18f9b43f224fb72e2a3030be80eadb8706ab0c0610f9cd9ed2134745d4"}
Oct 30 22:45:02.024669 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.024621    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl" event={"ID":"78b13226-d659-4865-9627-024fd2ced764","Type":"ContainerStarted","Data":"be10524d95672d8b31ef45b37b26a3a31ae40ee998d496ace0794cb8e4500cb0"}
Oct 30 22:45:02.026401 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.026366    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw" event={"ID":"3d953f06-b11d-4115-8b64-3970e1639126","Type":"ContainerStarted","Data":"225d75b05c61206c97e73a87ca4f220f723340cf25efdfc542aa877db4e3b142"}
Oct 30 22:45:02.028651 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.028611    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-ww2qg" event={"ID":"62aab7d6-3e77-4389-8f85-ce60b5f90056","Type":"ContainerStarted","Data":"b30592dcf0abdacb3a87e6b16e01557d04b74357196013137639ae9d788d2a4a"}
Oct 30 22:45:02.032304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.032268    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr" event={"ID":"ec7b92fc-0bac-45c3-9adb-b0cd06b33198","Type":"ContainerStarted","Data":"666c13fdb6b72dfa6d17572f0b64c7523fc7395309ed6c57b13fcc5ed1804fe4"}
Oct 30 22:45:02.033944 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.033913    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d" event={"ID":"3530f8ff-0e20-4062-8939-aa465f065f24","Type":"ContainerStarted","Data":"1214b95bae1340ac830544cec060f8d0a1a8b83b94e5d1f1e8a59275e4246106"}
Oct 30 22:45:02.035349 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.035321    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj" event={"ID":"dfe9d27e-3639-4085-9f3d-93d8a9ec5063","Type":"ContainerStarted","Data":"94b55c841fcf7e59788cbdbc53c96d71179b5f370b0c0b4e9fe7b5e00320a04e"}
Oct 30 22:45:02.037665 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.037633    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g" event={"ID":"50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa","Type":"ContainerStarted","Data":"914e9c51c9489c73a51020b5b00750cc8645726462cccbe29d7f8a53ba13dad5"}
Oct 30 22:45:02.037822 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.037695    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g" event={"ID":"50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa","Type":"ContainerStarted","Data":"5b50ac9ce178e64ca4b1d16f45edcb8b817a8def87010e4a1d66ad34318ba22d"}
Oct 30 22:45:02.039136 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.039110    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f" event={"ID":"b83f41b5-12fa-4cd6-896c-5639ead76511","Type":"ContainerStarted","Data":"593c13867d0a170accb6f2eed7864cb10f8d456651628edd137dff3cb1443d2a"}
Oct 30 22:45:02.065029 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.064960    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8" podStartSLOduration=2.064943276 podStartE2EDuration="2.064943276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:02.061083959 +0000 UTC m=+2287.417450288" watchObservedRunningTime="2026-10-30 22:45:02.064943276 +0000 UTC m=+2287.421309593"
Oct 30 22:45:02.111992 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.111815    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g" podStartSLOduration=2.111794902 podStartE2EDuration="2.111794902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:02.106847899 +0000 UTC m=+2287.463214213" watchObservedRunningTime="2026-10-30 22:45:02.111794902 +0000 UTC m=+2287.468161219"
Oct 30 22:45:02.240140 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.240100    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:02.347464 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.347426    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2"]
Oct 30 22:45:02.352858 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.352832    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2"
Oct 30 22:45:02.373853 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.373810    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2"]
Oct 30 22:45:02.375867 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.375836    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-c511a3\""
Oct 30 22:45:02.376300 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.376280    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-c511a3-dockercfg-kcp5m\""
Oct 30 22:45:02.433295 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.433184    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:02.464381 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.464342    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg"]
Oct 30 22:45:02.475787 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.475752    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg"
Oct 30 22:45:02.478676 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.478643    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg"]
Oct 30 22:45:02.519936 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.519896    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d175091-47d2-43ad-b201-1b7c0d8b82a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2\" (UID: \"7d175091-47d2-43ad-b201-1b7c0d8b82a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2"
Oct 30 22:45:02.520238 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.520201    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c47jw\" (UniqueName: \"kubernetes.io/projected/7d175091-47d2-43ad-b201-1b7c0d8b82a9-kube-api-access-c47jw\") pod \"payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2\" (UID: \"7d175091-47d2-43ad-b201-1b7c0d8b82a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2"
Oct 30 22:45:02.675424 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.650861    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e935aad1-ec9a-45e4-b4d4-31d6b84950a1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg\" (UID: \"e935aad1-ec9a-45e4-b4d4-31d6b84950a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg"
Oct 30 22:45:02.686505 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.686468    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c47jw\" (UniqueName: \"kubernetes.io/projected/7d175091-47d2-43ad-b201-1b7c0d8b82a9-kube-api-access-c47jw\") pod \"payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2\" (UID: \"7d175091-47d2-43ad-b201-1b7c0d8b82a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2"
Oct 30 22:45:02.687121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.687093    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d175091-47d2-43ad-b201-1b7c0d8b82a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2\" (UID: \"7d175091-47d2-43ad-b201-1b7c0d8b82a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2"
Oct 30 22:45:02.725137 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.725110    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzx4f\" (UniqueName: \"kubernetes.io/projected/e935aad1-ec9a-45e4-b4d4-31d6b84950a1-kube-api-access-qzx4f\") pod \"payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg\" (UID: \"e935aad1-ec9a-45e4-b4d4-31d6b84950a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg"
Oct 30 22:45:02.725811 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.701262    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d175091-47d2-43ad-b201-1b7c0d8b82a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2\" (UID: \"7d175091-47d2-43ad-b201-1b7c0d8b82a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2"
Oct 30 22:45:02.762553 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.753978    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c47jw\" (UniqueName: \"kubernetes.io/projected/7d175091-47d2-43ad-b201-1b7c0d8b82a9-kube-api-access-c47jw\") pod \"payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2\" (UID: \"7d175091-47d2-43ad-b201-1b7c0d8b82a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2"
Oct 30 22:45:02.827292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.827252    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzx4f\" (UniqueName: \"kubernetes.io/projected/e935aad1-ec9a-45e4-b4d4-31d6b84950a1-kube-api-access-qzx4f\") pod \"payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg\" (UID: \"e935aad1-ec9a-45e4-b4d4-31d6b84950a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg"
Oct 30 22:45:02.827522 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.827504    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e935aad1-ec9a-45e4-b4d4-31d6b84950a1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg\" (UID: \"e935aad1-ec9a-45e4-b4d4-31d6b84950a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg"
Oct 30 22:45:02.828464 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.828439    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e935aad1-ec9a-45e4-b4d4-31d6b84950a1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg\" (UID: \"e935aad1-ec9a-45e4-b4d4-31d6b84950a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg"
Oct 30 22:45:02.849026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.848988    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzx4f\" (UniqueName: \"kubernetes.io/projected/e935aad1-ec9a-45e4-b4d4-31d6b84950a1-kube-api-access-qzx4f\") pod \"payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg\" (UID: \"e935aad1-ec9a-45e4-b4d4-31d6b84950a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg"
Oct 30 22:45:02.973727 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:02.973686    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2"
Oct 30 22:45:03.079965 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.079919    4889 generic.go:356] "Generic (PLEG): container finished" podID="b83f41b5-12fa-4cd6-896c-5639ead76511" containerID="c8652b82710e5c26c7bd26010c7bfdb8082beadad9cf22304a46246f5ad7c10c" exitCode=60
Oct 30 22:45:03.080214 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.080191    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f" event={"ID":"b83f41b5-12fa-4cd6-896c-5639ead76511","Type":"ContainerDied","Data":"c8652b82710e5c26c7bd26010c7bfdb8082beadad9cf22304a46246f5ad7c10c"}
Oct 30 22:45:03.080945 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.080918    4889 scope.go:122] "RemoveContainer" containerID="c8652b82710e5c26c7bd26010c7bfdb8082beadad9cf22304a46246f5ad7c10c"
Oct 30 22:45:03.083877 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.083833    4889 generic.go:356] "Generic (PLEG): container finished" podID="23d07912-2775-4414-9754-d3e16bdf69d2" containerID="54fd5e18f9b43f224fb72e2a3030be80eadb8706ab0c0610f9cd9ed2134745d4" exitCode=60
Oct 30 22:45:03.084016 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.083962    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8" event={"ID":"23d07912-2775-4414-9754-d3e16bdf69d2","Type":"ContainerDied","Data":"54fd5e18f9b43f224fb72e2a3030be80eadb8706ab0c0610f9cd9ed2134745d4"}
Oct 30 22:45:03.084462 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.084438    4889 scope.go:122] "RemoveContainer" containerID="54fd5e18f9b43f224fb72e2a3030be80eadb8706ab0c0610f9cd9ed2134745d4"
Oct 30 22:45:03.091094 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.091062    4889 generic.go:356] "Generic (PLEG): container finished" podID="78b13226-d659-4865-9627-024fd2ced764" containerID="db36bdc3a74161bec1afce31cc1d4e1a66cb333c3d733b7ad7a1e971c6c627f7" exitCode=60
Oct 30 22:45:03.091355 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.091139    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl" event={"ID":"78b13226-d659-4865-9627-024fd2ced764","Type":"ContainerDied","Data":"db36bdc3a74161bec1afce31cc1d4e1a66cb333c3d733b7ad7a1e971c6c627f7"}
Oct 30 22:45:03.091756 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.091731    4889 scope.go:122] "RemoveContainer" containerID="db36bdc3a74161bec1afce31cc1d4e1a66cb333c3d733b7ad7a1e971c6c627f7"
Oct 30 22:45:03.097175 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.097144    4889 generic.go:356] "Generic (PLEG): container finished" podID="3d953f06-b11d-4115-8b64-3970e1639126" containerID="1e33b0b688706719fe883de6d3574b445b6fc387c923d438a4bc8bbc42a44bc6" exitCode=60
Oct 30 22:45:03.097370 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.097344    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw" event={"ID":"3d953f06-b11d-4115-8b64-3970e1639126","Type":"ContainerDied","Data":"1e33b0b688706719fe883de6d3574b445b6fc387c923d438a4bc8bbc42a44bc6"}
Oct 30 22:45:03.097794 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.097767    4889 scope.go:122] "RemoveContainer" containerID="1e33b0b688706719fe883de6d3574b445b6fc387c923d438a4bc8bbc42a44bc6"
Oct 30 22:45:03.099645 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.099621    4889 generic.go:356] "Generic (PLEG): container finished" podID="62aab7d6-3e77-4389-8f85-ce60b5f90056" containerID="a4c13e6b2bef9d9a399105176625c77a555df9092bc8d7d6a2623def077b49ae" exitCode=60
Oct 30 22:45:03.100043 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.100023    4889 scope.go:122] "RemoveContainer" containerID="a4c13e6b2bef9d9a399105176625c77a555df9092bc8d7d6a2623def077b49ae"
Oct 30 22:45:03.100303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.100273    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-ww2qg" event={"ID":"62aab7d6-3e77-4389-8f85-ce60b5f90056","Type":"ContainerDied","Data":"a4c13e6b2bef9d9a399105176625c77a555df9092bc8d7d6a2623def077b49ae"}
Oct 30 22:45:03.101912 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.101887    4889 generic.go:356] "Generic (PLEG): container finished" podID="ec7b92fc-0bac-45c3-9adb-b0cd06b33198" containerID="e41cb1a6a15015c7d7452b4feb7c0bf43c701572260e0433b72c2d04153e9d5c" exitCode=60
Oct 30 22:45:03.102044 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.101962    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr" event={"ID":"ec7b92fc-0bac-45c3-9adb-b0cd06b33198","Type":"ContainerDied","Data":"e41cb1a6a15015c7d7452b4feb7c0bf43c701572260e0433b72c2d04153e9d5c"}
Oct 30 22:45:03.102298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.102281    4889 scope.go:122] "RemoveContainer" containerID="e41cb1a6a15015c7d7452b4feb7c0bf43c701572260e0433b72c2d04153e9d5c"
Oct 30 22:45:03.103934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.103905    4889 generic.go:356] "Generic (PLEG): container finished" podID="3530f8ff-0e20-4062-8939-aa465f065f24" containerID="64b54559f6e738f7de100fb6c07120fd47cae09130e31ecc2396b6b2745fa8f1" exitCode=60
Oct 30 22:45:03.104102 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.104070    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d" event={"ID":"3530f8ff-0e20-4062-8939-aa465f065f24","Type":"ContainerDied","Data":"64b54559f6e738f7de100fb6c07120fd47cae09130e31ecc2396b6b2745fa8f1"}
Oct 30 22:45:03.104492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.104421    4889 scope.go:122] "RemoveContainer" containerID="64b54559f6e738f7de100fb6c07120fd47cae09130e31ecc2396b6b2745fa8f1"
Oct 30 22:45:03.109047 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.108750    4889 generic.go:356] "Generic (PLEG): container finished" podID="dfe9d27e-3639-4085-9f3d-93d8a9ec5063" containerID="09b1765d4357f19572409fd7a2ae61d4db1c636717c3e9713d8dbbb49b1594ce" exitCode=60
Oct 30 22:45:03.109047 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.108860    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj" event={"ID":"dfe9d27e-3639-4085-9f3d-93d8a9ec5063","Type":"ContainerDied","Data":"09b1765d4357f19572409fd7a2ae61d4db1c636717c3e9713d8dbbb49b1594ce"}
Oct 30 22:45:03.109189 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.109064    4889 scope.go:122] "RemoveContainer" containerID="09b1765d4357f19572409fd7a2ae61d4db1c636717c3e9713d8dbbb49b1594ce"
Oct 30 22:45:03.112020 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.111989    4889 generic.go:356] "Generic (PLEG): container finished" podID="50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa" containerID="914e9c51c9489c73a51020b5b00750cc8645726462cccbe29d7f8a53ba13dad5" exitCode=60
Oct 30 22:45:03.112151 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.112051    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g" event={"ID":"50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa","Type":"ContainerDied","Data":"914e9c51c9489c73a51020b5b00750cc8645726462cccbe29d7f8a53ba13dad5"}
Oct 30 22:45:03.112534 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.112513    4889 scope.go:122] "RemoveContainer" containerID="914e9c51c9489c73a51020b5b00750cc8645726462cccbe29d7f8a53ba13dad5"
Oct 30 22:45:03.124509 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.124482    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg"
Oct 30 22:45:03.229987 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.225357    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:03.298614 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:45:03.298585    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d175091_47d2_43ad_b201_1b7c0d8b82a9.slice/crio-81e3ef97c83ed9569f7e05a0b9f5e010c6530f72dc1cbe70fc33622035e09be0 WatchSource:0}: Error finding container 81e3ef97c83ed9569f7e05a0b9f5e010c6530f72dc1cbe70fc33622035e09be0: Status 404 returned error can't find the container with id 81e3ef97c83ed9569f7e05a0b9f5e010c6530f72dc1cbe70fc33622035e09be0
Oct 30 22:45:03.302391 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.302368    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2"]
Oct 30 22:45:03.798944 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:03.798914    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg"]
Oct 30 22:45:03.813420 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:45:03.813368    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode935aad1_ec9a_45e4_b4d4_31d6b84950a1.slice/crio-b4b5eb5627ef5ef876cde108b7e82edb960c9bb5100ae7ff96bced9594c13aa8 WatchSource:0}: Error finding container b4b5eb5627ef5ef876cde108b7e82edb960c9bb5100ae7ff96bced9594c13aa8: Status 404 returned error can't find the container with id b4b5eb5627ef5ef876cde108b7e82edb960c9bb5100ae7ff96bced9594c13aa8
Oct 30 22:45:04.132010 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:04.131967    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8" event={"ID":"23d07912-2775-4414-9754-d3e16bdf69d2","Type":"ContainerStarted","Data":"34f6173b06f3e87a71ff1caed3ca579e48e5a7ecc452d5e7851c2f386d889624"}
Oct 30 22:45:04.135580 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:04.135550    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-ww2qg" event={"ID":"62aab7d6-3e77-4389-8f85-ce60b5f90056","Type":"ContainerStarted","Data":"715ca6458379fab74668a394df92a509d7dbfe0ceb57b58b5317f04887b0e85e"}
Oct 30 22:45:04.137802 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:04.137762    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg" event={"ID":"e935aad1-ec9a-45e4-b4d4-31d6b84950a1","Type":"ContainerStarted","Data":"b4b5eb5627ef5ef876cde108b7e82edb960c9bb5100ae7ff96bced9594c13aa8"}
Oct 30 22:45:04.140095 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:04.140027    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d" event={"ID":"3530f8ff-0e20-4062-8939-aa465f065f24","Type":"ContainerStarted","Data":"2c1886f127232d4e02d6bac83682d7ff0092feab3f309e8137e6af45418caa7f"}
Oct 30 22:45:04.142908 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:04.142873    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj" event={"ID":"dfe9d27e-3639-4085-9f3d-93d8a9ec5063","Type":"ContainerStarted","Data":"35aa425160189d5b2394af6e3b53fdbed575b5073aa4dd5213386edfc1507178"}
Oct 30 22:45:04.146381 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:04.146334    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g" event={"ID":"50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa","Type":"ContainerStarted","Data":"28dfb00a46cee283e33f3b1e0d6a63ff7b7eb17a2198b5e7824909a45cd413fb"}
Oct 30 22:45:04.149323 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:04.149290    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f" event={"ID":"b83f41b5-12fa-4cd6-896c-5639ead76511","Type":"ContainerStarted","Data":"68a9868100c5e92f795d3ecae88c100426b33ffda41d5c34202fe2e6421a4a51"}
Oct 30 22:45:04.151325 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:04.151298    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2" event={"ID":"7d175091-47d2-43ad-b201-1b7c0d8b82a9","Type":"ContainerStarted","Data":"6afbca25923bc23cabb77349037c6545edb90c4ce30a263ece94aeb7f32ef1ee"}
Oct 30 22:45:04.151480 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:04.151348    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2" event={"ID":"7d175091-47d2-43ad-b201-1b7c0d8b82a9","Type":"ContainerStarted","Data":"81e3ef97c83ed9569f7e05a0b9f5e010c6530f72dc1cbe70fc33622035e09be0"}
Oct 30 22:45:04.151480 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:04.151403    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2"
Oct 30 22:45:04.219725 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:04.219666    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f" podStartSLOduration=4.219647929 podStartE2EDuration="4.219647929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:04.213865458 +0000 UTC m=+2289.570231773" watchObservedRunningTime="2026-10-30 22:45:04.219647929 +0000 UTC m=+2289.576014256"
Oct 30 22:45:04.257443 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:04.257387    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-ww2qg" podStartSLOduration=4.257373193 podStartE2EDuration="4.257373193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:04.25575146 +0000 UTC m=+2289.612117776" watchObservedRunningTime="2026-10-30 22:45:04.257373193 +0000 UTC m=+2289.613739508"
Oct 30 22:45:04.342109 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:04.341989    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d" podStartSLOduration=4.341972495 podStartE2EDuration="4.341972495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:04.33966323 +0000 UTC m=+2289.696029545" watchObservedRunningTime="2026-10-30 22:45:04.341972495 +0000 UTC m=+2289.698338811"
Oct 30 22:45:04.385693 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:04.385627    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2" podStartSLOduration=2.38560887 podStartE2EDuration="2.38560887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:04.380919604 +0000 UTC m=+2289.737285918" watchObservedRunningTime="2026-10-30 22:45:04.38560887 +0000 UTC m=+2289.741975185"
Oct 30 22:45:04.430987 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:04.430931    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj" podStartSLOduration=4.430914311 podStartE2EDuration="4.430914311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:04.428037032 +0000 UTC m=+2289.784403361" watchObservedRunningTime="2026-10-30 22:45:04.430914311 +0000 UTC m=+2289.787280626"
Oct 30 22:45:05.020607 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.020569    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:05.164131 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.164038    4889 generic.go:356] "Generic (PLEG): container finished" podID="b83f41b5-12fa-4cd6-896c-5639ead76511" containerID="68a9868100c5e92f795d3ecae88c100426b33ffda41d5c34202fe2e6421a4a51" exitCode=60
Oct 30 22:45:05.164325 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.164117    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f" event={"ID":"b83f41b5-12fa-4cd6-896c-5639ead76511","Type":"ContainerDied","Data":"68a9868100c5e92f795d3ecae88c100426b33ffda41d5c34202fe2e6421a4a51"}
Oct 30 22:45:05.164325 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.164278    4889 scope.go:122] "RemoveContainer" containerID="c8652b82710e5c26c7bd26010c7bfdb8082beadad9cf22304a46246f5ad7c10c"
Oct 30 22:45:05.164703 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.164684    4889 scope.go:122] "RemoveContainer" containerID="68a9868100c5e92f795d3ecae88c100426b33ffda41d5c34202fe2e6421a4a51"
Oct 30 22:45:05.165182 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:05.165148    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f_odh-ai-gateway-infra(b83f41b5-12fa-4cd6-896c-5639ead76511)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f" podUID="b83f41b5-12fa-4cd6-896c-5639ead76511"
Oct 30 22:45:05.166853 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.166832    4889 generic.go:356] "Generic (PLEG): container finished" podID="23d07912-2775-4414-9754-d3e16bdf69d2" containerID="34f6173b06f3e87a71ff1caed3ca579e48e5a7ecc452d5e7851c2f386d889624" exitCode=60
Oct 30 22:45:05.166963 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.166916    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8" event={"ID":"23d07912-2775-4414-9754-d3e16bdf69d2","Type":"ContainerDied","Data":"34f6173b06f3e87a71ff1caed3ca579e48e5a7ecc452d5e7851c2f386d889624"}
Oct 30 22:45:05.167298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.167276    4889 scope.go:122] "RemoveContainer" containerID="34f6173b06f3e87a71ff1caed3ca579e48e5a7ecc452d5e7851c2f386d889624"
Oct 30 22:45:05.167705 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:05.167683    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8_odh-ai-gateway-infra(23d07912-2775-4414-9754-d3e16bdf69d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8" podUID="23d07912-2775-4414-9754-d3e16bdf69d2"
Oct 30 22:45:05.169710 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.169683    4889 generic.go:356] "Generic (PLEG): container finished" podID="78b13226-d659-4865-9627-024fd2ced764" containerID="6db2012b65dc2a528e7b1db0e53151c4d45b8c2c5c05d94615df859b991bf816" exitCode=60
Oct 30 22:45:05.169816 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.169729    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl" event={"ID":"78b13226-d659-4865-9627-024fd2ced764","Type":"ContainerDied","Data":"6db2012b65dc2a528e7b1db0e53151c4d45b8c2c5c05d94615df859b991bf816"}
Oct 30 22:45:05.170127 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.170108    4889 scope.go:122] "RemoveContainer" containerID="6db2012b65dc2a528e7b1db0e53151c4d45b8c2c5c05d94615df859b991bf816"
Oct 30 22:45:05.170605 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:05.170585    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl_odh-ai-gateway-infra(78b13226-d659-4865-9627-024fd2ced764)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl" podUID="78b13226-d659-4865-9627-024fd2ced764"
Oct 30 22:45:05.172032 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.172008    4889 generic.go:356] "Generic (PLEG): container finished" podID="3d953f06-b11d-4115-8b64-3970e1639126" containerID="e7fa8dcc3b07e06d22e50adcfbfba70ee82dfd5784cfa81da539a7bd881558d4" exitCode=60
Oct 30 22:45:05.172134 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.172117    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw" event={"ID":"3d953f06-b11d-4115-8b64-3970e1639126","Type":"ContainerDied","Data":"e7fa8dcc3b07e06d22e50adcfbfba70ee82dfd5784cfa81da539a7bd881558d4"}
Oct 30 22:45:05.172594 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.172575    4889 scope.go:122] "RemoveContainer" containerID="e7fa8dcc3b07e06d22e50adcfbfba70ee82dfd5784cfa81da539a7bd881558d4"
Oct 30 22:45:05.173038 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:05.173011    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw_odh-ai-gateway-infra(3d953f06-b11d-4115-8b64-3970e1639126)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw" podUID="3d953f06-b11d-4115-8b64-3970e1639126"
Oct 30 22:45:05.174495 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.174472    4889 generic.go:356] "Generic (PLEG): container finished" podID="62aab7d6-3e77-4389-8f85-ce60b5f90056" containerID="715ca6458379fab74668a394df92a509d7dbfe0ceb57b58b5317f04887b0e85e" exitCode=60
Oct 30 22:45:05.174605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.174553    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-ww2qg" event={"ID":"62aab7d6-3e77-4389-8f85-ce60b5f90056","Type":"ContainerDied","Data":"715ca6458379fab74668a394df92a509d7dbfe0ceb57b58b5317f04887b0e85e"}
Oct 30 22:45:05.174913 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.174898    4889 scope.go:122] "RemoveContainer" containerID="715ca6458379fab74668a394df92a509d7dbfe0ceb57b58b5317f04887b0e85e"
Oct 30 22:45:05.175382 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:05.175357    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-ww2qg_odh-ai-gateway-infra(62aab7d6-3e77-4389-8f85-ce60b5f90056)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-ww2qg" podUID="62aab7d6-3e77-4389-8f85-ce60b5f90056"
Oct 30 22:45:05.176829 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.176808    4889 generic.go:356] "Generic (PLEG): container finished" podID="ec7b92fc-0bac-45c3-9adb-b0cd06b33198" containerID="6300e9e64864ccaa22c9e8fb5bf31739fb5ad48961c2234090d666c67c1e19c7" exitCode=60
Oct 30 22:45:05.176943 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.176892    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr" event={"ID":"ec7b92fc-0bac-45c3-9adb-b0cd06b33198","Type":"ContainerDied","Data":"6300e9e64864ccaa22c9e8fb5bf31739fb5ad48961c2234090d666c67c1e19c7"}
Oct 30 22:45:05.177269 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.177250    4889 scope.go:122] "RemoveContainer" containerID="6300e9e64864ccaa22c9e8fb5bf31739fb5ad48961c2234090d666c67c1e19c7"
Oct 30 22:45:05.177837 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:05.177812    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr_odh-ai-gateway-infra(ec7b92fc-0bac-45c3-9adb-b0cd06b33198)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr" podUID="ec7b92fc-0bac-45c3-9adb-b0cd06b33198"
Oct 30 22:45:05.178960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.178930    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg" event={"ID":"e935aad1-ec9a-45e4-b4d4-31d6b84950a1","Type":"ContainerStarted","Data":"7aa54b2d5342eedd14e9e50f6079d7bc370bda3a857f40f4521a57e0c8c2de0c"}
Oct 30 22:45:05.179093 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.179029    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg"
Oct 30 22:45:05.181272 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.181245    4889 generic.go:356] "Generic (PLEG): container finished" podID="3530f8ff-0e20-4062-8939-aa465f065f24" containerID="2c1886f127232d4e02d6bac83682d7ff0092feab3f309e8137e6af45418caa7f" exitCode=60
Oct 30 22:45:05.181394 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.181345    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d" event={"ID":"3530f8ff-0e20-4062-8939-aa465f065f24","Type":"ContainerDied","Data":"2c1886f127232d4e02d6bac83682d7ff0092feab3f309e8137e6af45418caa7f"}
Oct 30 22:45:05.181638 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.181621    4889 scope.go:122] "RemoveContainer" containerID="2c1886f127232d4e02d6bac83682d7ff0092feab3f309e8137e6af45418caa7f"
Oct 30 22:45:05.182034 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:05.182013    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d_odh-ai-gateway-infra(3530f8ff-0e20-4062-8939-aa465f065f24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d" podUID="3530f8ff-0e20-4062-8939-aa465f065f24"
Oct 30 22:45:05.182320 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.182296    4889 scope.go:122] "RemoveContainer" containerID="54fd5e18f9b43f224fb72e2a3030be80eadb8706ab0c0610f9cd9ed2134745d4"
Oct 30 22:45:05.183666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.183637    4889 generic.go:356] "Generic (PLEG): container finished" podID="dfe9d27e-3639-4085-9f3d-93d8a9ec5063" containerID="35aa425160189d5b2394af6e3b53fdbed575b5073aa4dd5213386edfc1507178" exitCode=60
Oct 30 22:45:05.183778 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.183755    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj" event={"ID":"dfe9d27e-3639-4085-9f3d-93d8a9ec5063","Type":"ContainerDied","Data":"35aa425160189d5b2394af6e3b53fdbed575b5073aa4dd5213386edfc1507178"}
Oct 30 22:45:05.184028 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.184014    4889 scope.go:122] "RemoveContainer" containerID="35aa425160189d5b2394af6e3b53fdbed575b5073aa4dd5213386edfc1507178"
Oct 30 22:45:05.184445 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:05.184419    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj_odh-ai-gateway-infra(dfe9d27e-3639-4085-9f3d-93d8a9ec5063)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj" podUID="dfe9d27e-3639-4085-9f3d-93d8a9ec5063"
Oct 30 22:45:05.185898 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.185873    4889 generic.go:356] "Generic (PLEG): container finished" podID="50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa" containerID="28dfb00a46cee283e33f3b1e0d6a63ff7b7eb17a2198b5e7824909a45cd413fb" exitCode=60
Oct 30 22:45:05.185999 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.185943    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g" event={"ID":"50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa","Type":"ContainerDied","Data":"28dfb00a46cee283e33f3b1e0d6a63ff7b7eb17a2198b5e7824909a45cd413fb"}
Oct 30 22:45:05.186187 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.186173    4889 scope.go:122] "RemoveContainer" containerID="28dfb00a46cee283e33f3b1e0d6a63ff7b7eb17a2198b5e7824909a45cd413fb"
Oct 30 22:45:05.186551 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:05.186533    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g_odh-ai-gateway-infra(50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g" podUID="50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa"
Oct 30 22:45:05.199359 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.199333    4889 scope.go:122] "RemoveContainer" containerID="db36bdc3a74161bec1afce31cc1d4e1a66cb333c3d733b7ad7a1e971c6c627f7"
Oct 30 22:45:05.218125 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.218083    4889 scope.go:122] "RemoveContainer" containerID="1e33b0b688706719fe883de6d3574b445b6fc387c923d438a4bc8bbc42a44bc6"
Oct 30 22:45:05.234127 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.234101    4889 scope.go:122] "RemoveContainer" containerID="a4c13e6b2bef9d9a399105176625c77a555df9092bc8d7d6a2623def077b49ae"
Oct 30 22:45:05.251784 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.251764    4889 scope.go:122] "RemoveContainer" containerID="e41cb1a6a15015c7d7452b4feb7c0bf43c701572260e0433b72c2d04153e9d5c"
Oct 30 22:45:05.257374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.257322    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg" podStartSLOduration=3.257305112 podStartE2EDuration="3.257305112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:05.253665951 +0000 UTC m=+2290.610032281" watchObservedRunningTime="2026-10-30 22:45:05.257305112 +0000 UTC m=+2290.613671448"
Oct 30 22:45:05.268490 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.268469    4889 scope.go:122] "RemoveContainer" containerID="64b54559f6e738f7de100fb6c07120fd47cae09130e31ecc2396b6b2745fa8f1"
Oct 30 22:45:05.286766 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.286739    4889 scope.go:122] "RemoveContainer" containerID="09b1765d4357f19572409fd7a2ae61d4db1c636717c3e9713d8dbbb49b1594ce"
Oct 30 22:45:05.305205 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:05.305182    4889 scope.go:122] "RemoveContainer" containerID="914e9c51c9489c73a51020b5b00750cc8645726462cccbe29d7f8a53ba13dad5"
Oct 30 22:45:06.203897 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:06.203867    4889 scope.go:122] "RemoveContainer" containerID="715ca6458379fab74668a394df92a509d7dbfe0ceb57b58b5317f04887b0e85e"
Oct 30 22:45:06.204413 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:06.204286    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-ww2qg_odh-ai-gateway-infra(62aab7d6-3e77-4389-8f85-ce60b5f90056)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-ww2qg" podUID="62aab7d6-3e77-4389-8f85-ce60b5f90056"
Oct 30 22:45:06.205993 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:06.205968    4889 scope.go:122] "RemoveContainer" containerID="6300e9e64864ccaa22c9e8fb5bf31739fb5ad48961c2234090d666c67c1e19c7"
Oct 30 22:45:06.206400 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:06.206371    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr_odh-ai-gateway-infra(ec7b92fc-0bac-45c3-9adb-b0cd06b33198)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr" podUID="ec7b92fc-0bac-45c3-9adb-b0cd06b33198"
Oct 30 22:45:06.207778 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:06.207760    4889 scope.go:122] "RemoveContainer" containerID="2c1886f127232d4e02d6bac83682d7ff0092feab3f309e8137e6af45418caa7f"
Oct 30 22:45:06.208211 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:06.208183    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d_odh-ai-gateway-infra(3530f8ff-0e20-4062-8939-aa465f065f24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d" podUID="3530f8ff-0e20-4062-8939-aa465f065f24"
Oct 30 22:45:06.210167 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:06.210144    4889 scope.go:122] "RemoveContainer" containerID="35aa425160189d5b2394af6e3b53fdbed575b5073aa4dd5213386edfc1507178"
Oct 30 22:45:06.210511 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:06.210483    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj_odh-ai-gateway-infra(dfe9d27e-3639-4085-9f3d-93d8a9ec5063)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj" podUID="dfe9d27e-3639-4085-9f3d-93d8a9ec5063"
Oct 30 22:45:06.211902 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:06.211874    4889 scope.go:122] "RemoveContainer" containerID="28dfb00a46cee283e33f3b1e0d6a63ff7b7eb17a2198b5e7824909a45cd413fb"
Oct 30 22:45:06.212243 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:06.212202    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g_odh-ai-gateway-infra(50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g" podUID="50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa"
Oct 30 22:45:06.213848 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:06.213827    4889 scope.go:122] "RemoveContainer" containerID="68a9868100c5e92f795d3ecae88c100426b33ffda41d5c34202fe2e6421a4a51"
Oct 30 22:45:06.214178 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:06.214154    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f_odh-ai-gateway-infra(b83f41b5-12fa-4cd6-896c-5639ead76511)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f" podUID="b83f41b5-12fa-4cd6-896c-5639ead76511"
Oct 30 22:45:06.215802 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:06.215781    4889 scope.go:122] "RemoveContainer" containerID="34f6173b06f3e87a71ff1caed3ca579e48e5a7ecc452d5e7851c2f386d889624"
Oct 30 22:45:06.216160 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:06.216137    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8_odh-ai-gateway-infra(23d07912-2775-4414-9754-d3e16bdf69d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8" podUID="23d07912-2775-4414-9754-d3e16bdf69d2"
Oct 30 22:45:06.217690 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:06.217661    4889 scope.go:122] "RemoveContainer" containerID="6db2012b65dc2a528e7b1db0e53151c4d45b8c2c5c05d94615df859b991bf816"
Oct 30 22:45:06.218031 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:06.218008    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl_odh-ai-gateway-infra(78b13226-d659-4865-9627-024fd2ced764)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl" podUID="78b13226-d659-4865-9627-024fd2ced764"
Oct 30 22:45:06.219390 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:06.219371    4889 scope.go:122] "RemoveContainer" containerID="e7fa8dcc3b07e06d22e50adcfbfba70ee82dfd5784cfa81da539a7bd881558d4"
Oct 30 22:45:06.219650 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:06.219630    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw_odh-ai-gateway-infra(3d953f06-b11d-4115-8b64-3970e1639126)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw" podUID="3d953f06-b11d-4115-8b64-3970e1639126"
Oct 30 22:45:07.908065 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:07.908025    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:08.614314 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:08.614274    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:10.304522 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:10.304481    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:45:10.366054 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:10.366014    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:12.112972 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:12.112935    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:13.874293 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:13.874206    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:15.189527 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:15.189495    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2"
Oct 30 22:45:15.640576 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:15.640534    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:15.767901 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:15.767874    4889 scope.go:122] "RemoveContainer" containerID="6300e9e64864ccaa22c9e8fb5bf31739fb5ad48961c2234090d666c67c1e19c7"
Oct 30 22:45:15.768055 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:15.768041    4889 scope.go:122] "RemoveContainer" containerID="35aa425160189d5b2394af6e3b53fdbed575b5073aa4dd5213386edfc1507178"
Oct 30 22:45:15.768211 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:15.768174    4889 scope.go:122] "RemoveContainer" containerID="34f6173b06f3e87a71ff1caed3ca579e48e5a7ecc452d5e7851c2f386d889624"
Oct 30 22:45:16.224255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:16.224205    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg"
Oct 30 22:45:16.769354 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:16.767188    4889 scope.go:122] "RemoveContainer" containerID="2c1886f127232d4e02d6bac83682d7ff0092feab3f309e8137e6af45418caa7f"
Oct 30 22:45:16.769354 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:16.767404    4889 scope.go:122] "RemoveContainer" containerID="715ca6458379fab74668a394df92a509d7dbfe0ceb57b58b5317f04887b0e85e"
Oct 30 22:45:16.769354 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:16.767764    4889 scope.go:122] "RemoveContainer" containerID="68a9868100c5e92f795d3ecae88c100426b33ffda41d5c34202fe2e6421a4a51"
Oct 30 22:45:17.333457 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:17.333362    4889 generic.go:356] "Generic (PLEG): container finished" podID="ec7b92fc-0bac-45c3-9adb-b0cd06b33198" containerID="ce7c41f0bc4679e47c4d06ffdc923d078932c63e11f4f5be95805bd6f0d62080" exitCode=60
Oct 30 22:45:17.333457 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:17.333438    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr" event={"ID":"ec7b92fc-0bac-45c3-9adb-b0cd06b33198","Type":"ContainerDied","Data":"ce7c41f0bc4679e47c4d06ffdc923d078932c63e11f4f5be95805bd6f0d62080"}
Oct 30 22:45:17.333997 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:17.333548    4889 scope.go:122] "RemoveContainer" containerID="6300e9e64864ccaa22c9e8fb5bf31739fb5ad48961c2234090d666c67c1e19c7"
Oct 30 22:45:17.334094 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:17.334074    4889 scope.go:122] "RemoveContainer" containerID="ce7c41f0bc4679e47c4d06ffdc923d078932c63e11f4f5be95805bd6f0d62080"
Oct 30 22:45:17.334572 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:17.334543    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr_odh-ai-gateway-infra(ec7b92fc-0bac-45c3-9adb-b0cd06b33198)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr" podUID="ec7b92fc-0bac-45c3-9adb-b0cd06b33198"
Oct 30 22:45:17.335923 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:17.335896    4889 generic.go:356] "Generic (PLEG): container finished" podID="dfe9d27e-3639-4085-9f3d-93d8a9ec5063" containerID="99dc2539c50cf967f124624089c6418db5b8c11fd809051ef0c175336375be54" exitCode=60
Oct 30 22:45:17.336027 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:17.335966    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj" event={"ID":"dfe9d27e-3639-4085-9f3d-93d8a9ec5063","Type":"ContainerDied","Data":"99dc2539c50cf967f124624089c6418db5b8c11fd809051ef0c175336375be54"}
Oct 30 22:45:17.336480 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:17.336462    4889 scope.go:122] "RemoveContainer" containerID="99dc2539c50cf967f124624089c6418db5b8c11fd809051ef0c175336375be54"
Oct 30 22:45:17.336931 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:17.336907    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj_odh-ai-gateway-infra(dfe9d27e-3639-4085-9f3d-93d8a9ec5063)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj" podUID="dfe9d27e-3639-4085-9f3d-93d8a9ec5063"
Oct 30 22:45:17.338525 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:17.338502    4889 generic.go:356] "Generic (PLEG): container finished" podID="23d07912-2775-4414-9754-d3e16bdf69d2" containerID="433f82cac3843c601eff791010a952443513debcb94c1e0bfe6db39e59af7917" exitCode=60
Oct 30 22:45:17.338628 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:17.338528    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8" event={"ID":"23d07912-2775-4414-9754-d3e16bdf69d2","Type":"ContainerDied","Data":"433f82cac3843c601eff791010a952443513debcb94c1e0bfe6db39e59af7917"}
Oct 30 22:45:17.338989 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:17.338971    4889 scope.go:122] "RemoveContainer" containerID="433f82cac3843c601eff791010a952443513debcb94c1e0bfe6db39e59af7917"
Oct 30 22:45:17.339443 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:17.339421    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8_odh-ai-gateway-infra(23d07912-2775-4414-9754-d3e16bdf69d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8" podUID="23d07912-2775-4414-9754-d3e16bdf69d2"
Oct 30 22:45:17.349795 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:17.349764    4889 scope.go:122] "RemoveContainer" containerID="35aa425160189d5b2394af6e3b53fdbed575b5073aa4dd5213386edfc1507178"
Oct 30 22:45:17.365791 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:17.365769    4889 scope.go:122] "RemoveContainer" containerID="34f6173b06f3e87a71ff1caed3ca579e48e5a7ecc452d5e7851c2f386d889624"
Oct 30 22:45:17.497131 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:17.497093    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:17.767011 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:17.766973    4889 scope.go:122] "RemoveContainer" containerID="e7fa8dcc3b07e06d22e50adcfbfba70ee82dfd5784cfa81da539a7bd881558d4"
Oct 30 22:45:17.767395 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:17.767373    4889 scope.go:122] "RemoveContainer" containerID="28dfb00a46cee283e33f3b1e0d6a63ff7b7eb17a2198b5e7824909a45cd413fb"
Oct 30 22:45:17.767549 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:17.767530    4889 scope.go:122] "RemoveContainer" containerID="6db2012b65dc2a528e7b1db0e53151c4d45b8c2c5c05d94615df859b991bf816"
Oct 30 22:45:18.349750 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.349706    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw" event={"ID":"3d953f06-b11d-4115-8b64-3970e1639126","Type":"ContainerStarted","Data":"b10ef895fd9934a00fe494356f8fc9c474da8434e48351130f8bc5eb8fce0fad"}
Oct 30 22:45:18.351935 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.351895    4889 generic.go:356] "Generic (PLEG): container finished" podID="62aab7d6-3e77-4389-8f85-ce60b5f90056" containerID="8b2edb33c62f369d39489b097aa448d5de9ba76a48db8a6a7c78a35261fc8737" exitCode=60
Oct 30 22:45:18.352102 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.351995    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-ww2qg" event={"ID":"62aab7d6-3e77-4389-8f85-ce60b5f90056","Type":"ContainerDied","Data":"8b2edb33c62f369d39489b097aa448d5de9ba76a48db8a6a7c78a35261fc8737"}
Oct 30 22:45:18.352102 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.352066    4889 scope.go:122] "RemoveContainer" containerID="715ca6458379fab74668a394df92a509d7dbfe0ceb57b58b5317f04887b0e85e"
Oct 30 22:45:18.352654 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.352621    4889 scope.go:122] "RemoveContainer" containerID="8b2edb33c62f369d39489b097aa448d5de9ba76a48db8a6a7c78a35261fc8737"
Oct 30 22:45:18.353142 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:18.353110    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-ww2qg_odh-ai-gateway-infra(62aab7d6-3e77-4389-8f85-ce60b5f90056)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-ww2qg" podUID="62aab7d6-3e77-4389-8f85-ce60b5f90056"
Oct 30 22:45:18.356153 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.356120    4889 generic.go:356] "Generic (PLEG): container finished" podID="3530f8ff-0e20-4062-8939-aa465f065f24" containerID="deaa7e7064e458779785be1cc3402253025bb2c0f2c0944215a870432a976eb6" exitCode=60
Oct 30 22:45:18.356341 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.356189    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d" event={"ID":"3530f8ff-0e20-4062-8939-aa465f065f24","Type":"ContainerDied","Data":"deaa7e7064e458779785be1cc3402253025bb2c0f2c0944215a870432a976eb6"}
Oct 30 22:45:18.356810 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.356786    4889 scope.go:122] "RemoveContainer" containerID="deaa7e7064e458779785be1cc3402253025bb2c0f2c0944215a870432a976eb6"
Oct 30 22:45:18.357317 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:18.357292    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d_odh-ai-gateway-infra(3530f8ff-0e20-4062-8939-aa465f065f24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d" podUID="3530f8ff-0e20-4062-8939-aa465f065f24"
Oct 30 22:45:18.361094 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.361067    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g" event={"ID":"50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa","Type":"ContainerStarted","Data":"e03b37239b38d10b4248ce3fb6f6a6f712c5f2db28430085299097bb8a3d714c"}
Oct 30 22:45:18.363394 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.363369    4889 generic.go:356] "Generic (PLEG): container finished" podID="b83f41b5-12fa-4cd6-896c-5639ead76511" containerID="5c5bb07f1a8235296bdf3f948b92777e984ad2fb08839e53828ae9c85d6d85bf" exitCode=60
Oct 30 22:45:18.363510 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.363447    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f" event={"ID":"b83f41b5-12fa-4cd6-896c-5639ead76511","Type":"ContainerDied","Data":"5c5bb07f1a8235296bdf3f948b92777e984ad2fb08839e53828ae9c85d6d85bf"}
Oct 30 22:45:18.363878 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.363863    4889 scope.go:122] "RemoveContainer" containerID="5c5bb07f1a8235296bdf3f948b92777e984ad2fb08839e53828ae9c85d6d85bf"
Oct 30 22:45:18.364311 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:18.364287    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f_odh-ai-gateway-infra(b83f41b5-12fa-4cd6-896c-5639ead76511)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f" podUID="b83f41b5-12fa-4cd6-896c-5639ead76511"
Oct 30 22:45:18.370785 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.370724    4889 scope.go:122] "RemoveContainer" containerID="2c1886f127232d4e02d6bac83682d7ff0092feab3f309e8137e6af45418caa7f"
Oct 30 22:45:18.388755 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.388725    4889 scope.go:122] "RemoveContainer" containerID="68a9868100c5e92f795d3ecae88c100426b33ffda41d5c34202fe2e6421a4a51"
Oct 30 22:45:18.389461 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.389437    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr"]
Oct 30 22:45:18.398709 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.398641    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw" podStartSLOduration=18.398620555 podStartE2EDuration="18.398620555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:18.392870383 +0000 UTC m=+2303.749236698" watchObservedRunningTime="2026-10-30 22:45:18.398620555 +0000 UTC m=+2303.754986899"
Oct 30 22:45:18.431834 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.431799    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8"]
Oct 30 22:45:18.476753 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.476710    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj"]
Oct 30 22:45:18.670923 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.670888    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr"
Oct 30 22:45:18.738986 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.738958    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec7b92fc-0bac-45c3-9adb-b0cd06b33198-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7b92fc-0bac-45c3-9adb-b0cd06b33198-maas-api-service-ca\") pod \"ec7b92fc-0bac-45c3-9adb-b0cd06b33198\" (UID: \"ec7b92fc-0bac-45c3-9adb-b0cd06b33198\") "
Oct 30 22:45:18.739150 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.739123    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec7b92fc-0bac-45c3-9adb-b0cd06b33198-kube-api-access-2csbg\" (UniqueName: \"kubernetes.io/projected/ec7b92fc-0bac-45c3-9adb-b0cd06b33198-kube-api-access-2csbg\") pod \"ec7b92fc-0bac-45c3-9adb-b0cd06b33198\" (UID: \"ec7b92fc-0bac-45c3-9adb-b0cd06b33198\") "
Oct 30 22:45:18.739437 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.739409    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec7b92fc-0bac-45c3-9adb-b0cd06b33198-maas-api-service-ca" pod "ec7b92fc-0bac-45c3-9adb-b0cd06b33198" (UID: "ec7b92fc-0bac-45c3-9adb-b0cd06b33198"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:18.739975 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.739943    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7b92fc-0bac-45c3-9adb-b0cd06b33198-maas-api-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:45:18.742356 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.742326    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec7b92fc-0bac-45c3-9adb-b0cd06b33198-kube-api-access-2csbg" pod "ec7b92fc-0bac-45c3-9adb-b0cd06b33198" (UID: "ec7b92fc-0bac-45c3-9adb-b0cd06b33198"). InnerVolumeSpecName "kube-api-access-2csbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:18.750410 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.750381    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8"
Oct 30 22:45:18.802034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.802002    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj"
Oct 30 22:45:18.841503 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.841465    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d07912-2775-4414-9754-d3e16bdf69d2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d07912-2775-4414-9754-d3e16bdf69d2-maas-api-service-ca\") pod \"23d07912-2775-4414-9754-d3e16bdf69d2\" (UID: \"23d07912-2775-4414-9754-d3e16bdf69d2\") "
Oct 30 22:45:18.841666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.841588    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d07912-2775-4414-9754-d3e16bdf69d2-kube-api-access-9527p\" (UniqueName: \"kubernetes.io/projected/23d07912-2775-4414-9754-d3e16bdf69d2-kube-api-access-9527p\") pod \"23d07912-2775-4414-9754-d3e16bdf69d2\" (UID: \"23d07912-2775-4414-9754-d3e16bdf69d2\") "
Oct 30 22:45:18.843333 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.842971    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2csbg\" (UniqueName: \"kubernetes.io/projected/ec7b92fc-0bac-45c3-9adb-b0cd06b33198-kube-api-access-2csbg\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:45:18.846526 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.843799    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d07912-2775-4414-9754-d3e16bdf69d2-maas-api-service-ca" pod "23d07912-2775-4414-9754-d3e16bdf69d2" (UID: "23d07912-2775-4414-9754-d3e16bdf69d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:18.852396 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.852353    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d07912-2775-4414-9754-d3e16bdf69d2-kube-api-access-9527p" pod "23d07912-2775-4414-9754-d3e16bdf69d2" (UID: "23d07912-2775-4414-9754-d3e16bdf69d2"). InnerVolumeSpecName "kube-api-access-9527p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:18.944439 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.944033    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfe9d27e-3639-4085-9f3d-93d8a9ec5063-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe9d27e-3639-4085-9f3d-93d8a9ec5063-maas-api-service-ca\") pod \"dfe9d27e-3639-4085-9f3d-93d8a9ec5063\" (UID: \"dfe9d27e-3639-4085-9f3d-93d8a9ec5063\") "
Oct 30 22:45:18.944439 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.944157    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfe9d27e-3639-4085-9f3d-93d8a9ec5063-kube-api-access-q49gh\" (UniqueName: \"kubernetes.io/projected/dfe9d27e-3639-4085-9f3d-93d8a9ec5063-kube-api-access-q49gh\") pod \"dfe9d27e-3639-4085-9f3d-93d8a9ec5063\" (UID: \"dfe9d27e-3639-4085-9f3d-93d8a9ec5063\") "
Oct 30 22:45:18.944704 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.944451    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfe9d27e-3639-4085-9f3d-93d8a9ec5063-maas-api-service-ca" pod "dfe9d27e-3639-4085-9f3d-93d8a9ec5063" (UID: "dfe9d27e-3639-4085-9f3d-93d8a9ec5063"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:18.945085 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.945059    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9527p\" (UniqueName: \"kubernetes.io/projected/23d07912-2775-4414-9754-d3e16bdf69d2-kube-api-access-9527p\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:45:18.945085 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.945081    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe9d27e-3639-4085-9f3d-93d8a9ec5063-maas-api-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:45:18.945310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.945093    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d07912-2775-4414-9754-d3e16bdf69d2-maas-api-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:45:18.946767 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:18.946739    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe9d27e-3639-4085-9f3d-93d8a9ec5063-kube-api-access-q49gh" pod "dfe9d27e-3639-4085-9f3d-93d8a9ec5063" (UID: "dfe9d27e-3639-4085-9f3d-93d8a9ec5063"). InnerVolumeSpecName "kube-api-access-q49gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:19.046797 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.046759    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q49gh\" (UniqueName: \"kubernetes.io/projected/dfe9d27e-3639-4085-9f3d-93d8a9ec5063-kube-api-access-q49gh\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:45:19.326480 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.326442    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:19.381664 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.381577    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr" event={"ID":"ec7b92fc-0bac-45c3-9adb-b0cd06b33198","Type":"ContainerDied","Data":"666c13fdb6b72dfa6d17572f0b64c7523fc7395309ed6c57b13fcc5ed1804fe4"}
Oct 30 22:45:19.381664 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.381641    4889 scope.go:122] "RemoveContainer" containerID="ce7c41f0bc4679e47c4d06ffdc923d078932c63e11f4f5be95805bd6f0d62080"
Oct 30 22:45:19.382087 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.381715    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr"
Oct 30 22:45:19.388357 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.388319    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj" event={"ID":"dfe9d27e-3639-4085-9f3d-93d8a9ec5063","Type":"ContainerDied","Data":"94b55c841fcf7e59788cbdbc53c96d71179b5f370b0c0b4e9fe7b5e00320a04e"}
Oct 30 22:45:19.388659 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.388627    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj"
Oct 30 22:45:19.412150 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.412119    4889 scope.go:122] "RemoveContainer" containerID="99dc2539c50cf967f124624089c6418db5b8c11fd809051ef0c175336375be54"
Oct 30 22:45:19.413400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.413366    4889 generic.go:356] "Generic (PLEG): container finished" podID="50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa" containerID="e03b37239b38d10b4248ce3fb6f6a6f712c5f2db28430085299097bb8a3d714c" exitCode=60
Oct 30 22:45:19.413573 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.413549    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g" event={"ID":"50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa","Type":"ContainerDied","Data":"e03b37239b38d10b4248ce3fb6f6a6f712c5f2db28430085299097bb8a3d714c"}
Oct 30 22:45:19.413656 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.413636    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw"]
Oct 30 22:45:19.414138 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.414110    4889 scope.go:122] "RemoveContainer" containerID="e03b37239b38d10b4248ce3fb6f6a6f712c5f2db28430085299097bb8a3d714c"
Oct 30 22:45:19.423548 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:19.416754    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g_odh-ai-gateway-infra(50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g" podUID="50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa"
Oct 30 22:45:19.429631 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.429588    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8" event={"ID":"23d07912-2775-4414-9754-d3e16bdf69d2","Type":"ContainerDied","Data":"859b34ebd536dec298e03ccc442edc11c86b257bdc9d3c35b70e5ee0885ea9c4"}
Oct 30 22:45:19.430045 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.430022    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8"
Oct 30 22:45:19.440105 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.440059    4889 generic.go:356] "Generic (PLEG): container finished" podID="78b13226-d659-4865-9627-024fd2ced764" containerID="9e8d409aa3ccff0065a7c9332a03ac0c3c715ea77b289516c33717f6b4b8980f" exitCode=60
Oct 30 22:45:19.440352 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.440201    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl" event={"ID":"78b13226-d659-4865-9627-024fd2ced764","Type":"ContainerDied","Data":"9e8d409aa3ccff0065a7c9332a03ac0c3c715ea77b289516c33717f6b4b8980f"}
Oct 30 22:45:19.440879 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.440850    4889 scope.go:122] "RemoveContainer" containerID="9e8d409aa3ccff0065a7c9332a03ac0c3c715ea77b289516c33717f6b4b8980f"
Oct 30 22:45:19.441628 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:19.441545    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl_odh-ai-gateway-infra(78b13226-d659-4865-9627-024fd2ced764)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl" podUID="78b13226-d659-4865-9627-024fd2ced764"
Oct 30 22:45:19.446511 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.446488    4889 scope.go:122] "RemoveContainer" containerID="28dfb00a46cee283e33f3b1e0d6a63ff7b7eb17a2198b5e7824909a45cd413fb"
Oct 30 22:45:19.450800 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.449275    4889 generic.go:356] "Generic (PLEG): container finished" podID="3d953f06-b11d-4115-8b64-3970e1639126" containerID="b10ef895fd9934a00fe494356f8fc9c474da8434e48351130f8bc5eb8fce0fad" exitCode=60
Oct 30 22:45:19.450800 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.449340    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw" event={"ID":"3d953f06-b11d-4115-8b64-3970e1639126","Type":"ContainerDied","Data":"b10ef895fd9934a00fe494356f8fc9c474da8434e48351130f8bc5eb8fce0fad"}
Oct 30 22:45:19.450800 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.450303    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr"]
Oct 30 22:45:19.465110 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.464607    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f"]
Oct 30 22:45:19.471483 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.471363    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890005-6kwbr"]
Oct 30 22:45:19.487769 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.487720    4889 scope.go:122] "RemoveContainer" containerID="433f82cac3843c601eff791010a952443513debcb94c1e0bfe6db39e59af7917"
Oct 30 22:45:19.498498 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.498463    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-ww2qg"]
Oct 30 22:45:19.504536 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.504492    4889 scope.go:122] "RemoveContainer" containerID="6db2012b65dc2a528e7b1db0e53151c4d45b8c2c5c05d94615df859b991bf816"
Oct 30 22:45:19.526433 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.526409    4889 scope.go:122] "RemoveContainer" containerID="e7fa8dcc3b07e06d22e50adcfbfba70ee82dfd5784cfa81da539a7bd881558d4"
Oct 30 22:45:19.586668 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.586629    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g"]
Oct 30 22:45:19.649405 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.649322    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d"]
Oct 30 22:45:19.658328 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.657883    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8"]
Oct 30 22:45:19.666613 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.663610    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw"
Oct 30 22:45:19.666613 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.663773    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890005-kjkz8"]
Oct 30 22:45:19.726006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.725333    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f"
Oct 30 22:45:19.745843 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.745698    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj"]
Oct 30 22:45:19.755788 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.753175    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890005-rphcj"]
Oct 30 22:45:19.772043 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.772001    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d953f06-b11d-4115-8b64-3970e1639126-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d953f06-b11d-4115-8b64-3970e1639126-maas-api-service-ca\") pod \"3d953f06-b11d-4115-8b64-3970e1639126\" (UID: \"3d953f06-b11d-4115-8b64-3970e1639126\") "
Oct 30 22:45:19.772313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.772157    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d953f06-b11d-4115-8b64-3970e1639126-kube-api-access-rtj4r\" (UniqueName: \"kubernetes.io/projected/3d953f06-b11d-4115-8b64-3970e1639126-kube-api-access-rtj4r\") pod \"3d953f06-b11d-4115-8b64-3970e1639126\" (UID: \"3d953f06-b11d-4115-8b64-3970e1639126\") "
Oct 30 22:45:19.772527 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.772498    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d953f06-b11d-4115-8b64-3970e1639126-maas-api-service-ca" pod "3d953f06-b11d-4115-8b64-3970e1639126" (UID: "3d953f06-b11d-4115-8b64-3970e1639126"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:19.778650 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.778465    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d953f06-b11d-4115-8b64-3970e1639126-maas-api-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:45:19.781289 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.781008    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d953f06-b11d-4115-8b64-3970e1639126-kube-api-access-rtj4r" pod "3d953f06-b11d-4115-8b64-3970e1639126" (UID: "3d953f06-b11d-4115-8b64-3970e1639126"). InnerVolumeSpecName "kube-api-access-rtj4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:19.806460 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.806426    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-ww2qg"
Oct 30 22:45:19.879569 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.879184    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b83f41b5-12fa-4cd6-896c-5639ead76511-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b83f41b5-12fa-4cd6-896c-5639ead76511-maas-api-service-ca\") pod \"b83f41b5-12fa-4cd6-896c-5639ead76511\" (UID: \"b83f41b5-12fa-4cd6-896c-5639ead76511\") "
Oct 30 22:45:19.879569 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.879458    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b83f41b5-12fa-4cd6-896c-5639ead76511-kube-api-access-kwbkp\" (UniqueName: \"kubernetes.io/projected/b83f41b5-12fa-4cd6-896c-5639ead76511-kube-api-access-kwbkp\") pod \"b83f41b5-12fa-4cd6-896c-5639ead76511\" (UID: \"b83f41b5-12fa-4cd6-896c-5639ead76511\") "
Oct 30 22:45:19.880380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.879990    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b83f41b5-12fa-4cd6-896c-5639ead76511-maas-api-service-ca" pod "b83f41b5-12fa-4cd6-896c-5639ead76511" (UID: "b83f41b5-12fa-4cd6-896c-5639ead76511"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:19.880380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.880054    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtj4r\" (UniqueName: \"kubernetes.io/projected/3d953f06-b11d-4115-8b64-3970e1639126-kube-api-access-rtj4r\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:45:19.883289 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.883256    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b83f41b5-12fa-4cd6-896c-5639ead76511-kube-api-access-kwbkp" pod "b83f41b5-12fa-4cd6-896c-5639ead76511" (UID: "b83f41b5-12fa-4cd6-896c-5639ead76511"). InnerVolumeSpecName "kube-api-access-kwbkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:19.948932 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.948870    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d"
Oct 30 22:45:19.982219 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.982168    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62aab7d6-3e77-4389-8f85-ce60b5f90056-kube-api-access-tqjts\" (UniqueName: \"kubernetes.io/projected/62aab7d6-3e77-4389-8f85-ce60b5f90056-kube-api-access-tqjts\") pod \"62aab7d6-3e77-4389-8f85-ce60b5f90056\" (UID: \"62aab7d6-3e77-4389-8f85-ce60b5f90056\") "
Oct 30 22:45:19.982607 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.982546    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62aab7d6-3e77-4389-8f85-ce60b5f90056-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62aab7d6-3e77-4389-8f85-ce60b5f90056-maas-api-service-ca\") pod \"62aab7d6-3e77-4389-8f85-ce60b5f90056\" (UID: \"62aab7d6-3e77-4389-8f85-ce60b5f90056\") "
Oct 30 22:45:19.983972 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.983910    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b83f41b5-12fa-4cd6-896c-5639ead76511-maas-api-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:45:19.984152 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.984134    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwbkp\" (UniqueName: \"kubernetes.io/projected/b83f41b5-12fa-4cd6-896c-5639ead76511-kube-api-access-kwbkp\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:45:19.984757 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.984735    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62aab7d6-3e77-4389-8f85-ce60b5f90056-maas-api-service-ca" pod "62aab7d6-3e77-4389-8f85-ce60b5f90056" (UID: "62aab7d6-3e77-4389-8f85-ce60b5f90056"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:19.992567 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:19.992522    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62aab7d6-3e77-4389-8f85-ce60b5f90056-kube-api-access-tqjts" pod "62aab7d6-3e77-4389-8f85-ce60b5f90056" (UID: "62aab7d6-3e77-4389-8f85-ce60b5f90056"). InnerVolumeSpecName "kube-api-access-tqjts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:20.085137 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.085097    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3530f8ff-0e20-4062-8939-aa465f065f24-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3530f8ff-0e20-4062-8939-aa465f065f24-maas-api-service-ca\") pod \"3530f8ff-0e20-4062-8939-aa465f065f24\" (UID: \"3530f8ff-0e20-4062-8939-aa465f065f24\") "
Oct 30 22:45:20.085346 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.085268    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3530f8ff-0e20-4062-8939-aa465f065f24-kube-api-access-88t7r\" (UniqueName: \"kubernetes.io/projected/3530f8ff-0e20-4062-8939-aa465f065f24-kube-api-access-88t7r\") pod \"3530f8ff-0e20-4062-8939-aa465f065f24\" (UID: \"3530f8ff-0e20-4062-8939-aa465f065f24\") "
Oct 30 22:45:20.085588 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.085558    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3530f8ff-0e20-4062-8939-aa465f065f24-maas-api-service-ca" pod "3530f8ff-0e20-4062-8939-aa465f065f24" (UID: "3530f8ff-0e20-4062-8939-aa465f065f24"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:20.086042 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.086020    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqjts\" (UniqueName: \"kubernetes.io/projected/62aab7d6-3e77-4389-8f85-ce60b5f90056-kube-api-access-tqjts\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:45:20.086122 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.086047    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62aab7d6-3e77-4389-8f85-ce60b5f90056-maas-api-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:45:20.086122 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.086061    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3530f8ff-0e20-4062-8939-aa465f065f24-maas-api-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:45:20.087726 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.087703    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3530f8ff-0e20-4062-8939-aa465f065f24-kube-api-access-88t7r" pod "3530f8ff-0e20-4062-8939-aa465f065f24" (UID: "3530f8ff-0e20-4062-8939-aa465f065f24"). InnerVolumeSpecName "kube-api-access-88t7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:20.187731 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.187638    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88t7r\" (UniqueName: \"kubernetes.io/projected/3530f8ff-0e20-4062-8939-aa465f065f24-kube-api-access-88t7r\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:45:20.477625 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.477534    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f"
Oct 30 22:45:20.477625 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.477566    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f" event={"ID":"b83f41b5-12fa-4cd6-896c-5639ead76511","Type":"ContainerDied","Data":"593c13867d0a170accb6f2eed7864cb10f8d456651628edd137dff3cb1443d2a"}
Oct 30 22:45:20.478207 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.477647    4889 scope.go:122] "RemoveContainer" containerID="5c5bb07f1a8235296bdf3f948b92777e984ad2fb08839e53828ae9c85d6d85bf"
Oct 30 22:45:20.483646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.483620    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw"
Oct 30 22:45:20.483901 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.483647    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw" event={"ID":"3d953f06-b11d-4115-8b64-3970e1639126","Type":"ContainerDied","Data":"225d75b05c61206c97e73a87ca4f220f723340cf25efdfc542aa877db4e3b142"}
Oct 30 22:45:20.485452 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.485428    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-ww2qg"
Oct 30 22:45:20.490571 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.486856    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-ww2qg" event={"ID":"62aab7d6-3e77-4389-8f85-ce60b5f90056","Type":"ContainerDied","Data":"b30592dcf0abdacb3a87e6b16e01557d04b74357196013137639ae9d788d2a4a"}
Oct 30 22:45:20.491176 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.491146    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d" event={"ID":"3530f8ff-0e20-4062-8939-aa465f065f24","Type":"ContainerDied","Data":"1214b95bae1340ac830544cec060f8d0a1a8b83b94e5d1f1e8a59275e4246106"}
Oct 30 22:45:20.491330 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.491302    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d"
Oct 30 22:45:20.499424 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.499402    4889 scope.go:122] "RemoveContainer" containerID="b10ef895fd9934a00fe494356f8fc9c474da8434e48351130f8bc5eb8fce0fad"
Oct 30 22:45:20.514976 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.514954    4889 scope.go:122] "RemoveContainer" containerID="8b2edb33c62f369d39489b097aa448d5de9ba76a48db8a6a7c78a35261fc8737"
Oct 30 22:45:20.528976 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.528930    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl"]
Oct 30 22:45:20.530568 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.530465    4889 scope.go:122] "RemoveContainer" containerID="deaa7e7064e458779785be1cc3402253025bb2c0f2c0944215a870432a976eb6"
Oct 30 22:45:20.569068 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.568883    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f"]
Oct 30 22:45:20.575095 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.575061    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f0bb45-29890005-f5s9f"]
Oct 30 22:45:20.659838 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.659806    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw"]
Oct 30 22:45:20.667305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.667262    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890005-k24qw"]
Oct 30 22:45:20.687577 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.687539    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g"
Oct 30 22:45:20.691443 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.691395    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-ww2qg"]
Oct 30 22:45:20.700491 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.700448    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-ww2qg"]
Oct 30 22:45:20.730418 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.730327    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl"
Oct 30 22:45:20.780522 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.780491    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d07912-2775-4414-9754-d3e16bdf69d2" path="/var/lib/kubelet/pods/23d07912-2775-4414-9754-d3e16bdf69d2/volumes"
Oct 30 22:45:20.781077 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.781052    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d953f06-b11d-4115-8b64-3970e1639126" path="/var/lib/kubelet/pods/3d953f06-b11d-4115-8b64-3970e1639126/volumes"
Oct 30 22:45:20.781526 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.781504    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62aab7d6-3e77-4389-8f85-ce60b5f90056" path="/var/lib/kubelet/pods/62aab7d6-3e77-4389-8f85-ce60b5f90056/volumes"
Oct 30 22:45:20.782306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.782286    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b83f41b5-12fa-4cd6-896c-5639ead76511" path="/var/lib/kubelet/pods/b83f41b5-12fa-4cd6-896c-5639ead76511/volumes"
Oct 30 22:45:20.782777 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.782757    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfe9d27e-3639-4085-9f3d-93d8a9ec5063" path="/var/lib/kubelet/pods/dfe9d27e-3639-4085-9f3d-93d8a9ec5063/volumes"
Oct 30 22:45:20.783702 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.783682    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec7b92fc-0bac-45c3-9adb-b0cd06b33198" path="/var/lib/kubelet/pods/ec7b92fc-0bac-45c3-9adb-b0cd06b33198/volumes"
Oct 30 22:45:20.787907 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.787880    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d"]
Oct 30 22:45:20.788057 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.787929    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-867f12-29890005-4q52d"]
Oct 30 22:45:20.803577 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.803540    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa-maas-api-service-ca\") pod \"50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa\" (UID: \"50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa\") "
Oct 30 22:45:20.803731 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.803711    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa-kube-api-access-8cw7q\" (UniqueName: \"kubernetes.io/projected/50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa-kube-api-access-8cw7q\") pod \"50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa\" (UID: \"50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa\") "
Oct 30 22:45:20.803948 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.803924    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa-maas-api-service-ca" pod "50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa" (UID: "50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:20.804425 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.804368    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa-maas-api-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:45:20.806103 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.806080    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa-kube-api-access-8cw7q" pod "50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa" (UID: "50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa"). InnerVolumeSpecName "kube-api-access-8cw7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:20.905563 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.905518    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78b13226-d659-4865-9627-024fd2ced764-kube-api-access-wrsn6\" (UniqueName: \"kubernetes.io/projected/78b13226-d659-4865-9627-024fd2ced764-kube-api-access-wrsn6\") pod \"78b13226-d659-4865-9627-024fd2ced764\" (UID: \"78b13226-d659-4865-9627-024fd2ced764\") "
Oct 30 22:45:20.905563 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.905589    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78b13226-d659-4865-9627-024fd2ced764-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78b13226-d659-4865-9627-024fd2ced764-maas-api-service-ca\") pod \"78b13226-d659-4865-9627-024fd2ced764\" (UID: \"78b13226-d659-4865-9627-024fd2ced764\") "
Oct 30 22:45:20.906650 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.906620    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cw7q\" (UniqueName: \"kubernetes.io/projected/50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa-kube-api-access-8cw7q\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:45:20.906794 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.906745    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78b13226-d659-4865-9627-024fd2ced764-maas-api-service-ca" pod "78b13226-d659-4865-9627-024fd2ced764" (UID: "78b13226-d659-4865-9627-024fd2ced764"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:20.914566 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:20.914510    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b13226-d659-4865-9627-024fd2ced764-kube-api-access-wrsn6" pod "78b13226-d659-4865-9627-024fd2ced764" (UID: "78b13226-d659-4865-9627-024fd2ced764"). InnerVolumeSpecName "kube-api-access-wrsn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:21.008122 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:21.008077    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrsn6\" (UniqueName: \"kubernetes.io/projected/78b13226-d659-4865-9627-024fd2ced764-kube-api-access-wrsn6\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:45:21.008122 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:21.008114    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78b13226-d659-4865-9627-024fd2ced764-maas-api-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:45:21.126608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:21.126566    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:21.514165 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:21.514131    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g"
Oct 30 22:45:21.514165 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:21.514144    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g" event={"ID":"50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa","Type":"ContainerDied","Data":"5b50ac9ce178e64ca4b1d16f45edcb8b817a8def87010e4a1d66ad34318ba22d"}
Oct 30 22:45:21.514745 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:21.514305    4889 scope.go:122] "RemoveContainer" containerID="e03b37239b38d10b4248ce3fb6f6a6f712c5f2db28430085299097bb8a3d714c"
Oct 30 22:45:21.517519 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:21.517492    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl"
Oct 30 22:45:21.517662 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:21.517492    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl" event={"ID":"78b13226-d659-4865-9627-024fd2ced764","Type":"ContainerDied","Data":"be10524d95672d8b31ef45b37b26a3a31ae40ee998d496ace0794cb8e4500cb0"}
Oct 30 22:45:21.529050 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:21.529023    4889 scope.go:122] "RemoveContainer" containerID="9e8d409aa3ccff0065a7c9332a03ac0c3c715ea77b289516c33717f6b4b8980f"
Oct 30 22:45:21.584370 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:21.584336    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl"]
Oct 30 22:45:21.602898 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:21.602861    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890005-bqfhl"]
Oct 30 22:45:21.645746 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:21.645714    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g"]
Oct 30 22:45:21.652060 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:21.652022    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890005-cxb9g"]
Oct 30 22:45:22.782198 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:22.782152    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3530f8ff-0e20-4062-8939-aa465f065f24" path="/var/lib/kubelet/pods/3530f8ff-0e20-4062-8939-aa465f065f24/volumes"
Oct 30 22:45:22.782874 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:22.782854    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa" path="/var/lib/kubelet/pods/50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa/volumes"
Oct 30 22:45:22.783465 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:22.783442    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78b13226-d659-4865-9627-024fd2ced764" path="/var/lib/kubelet/pods/78b13226-d659-4865-9627-024fd2ced764/volumes"
Oct 30 22:45:23.053422 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:23.053291    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:24.816390 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:24.816349    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:26.684252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:26.683979    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:28.563244 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:28.562630    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:30.325323 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:30.315774    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:32.036842 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:32.036802    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:33.733476 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:33.733442    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:35.535142 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:35.535104    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:37.328168 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:37.328127    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:39.115806 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:39.115768    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:40.165885 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:40.165844    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:40.307066 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:45:40.307025    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:45:40.709296 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:40.709187    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:42.408437 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:42.408386    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:44.132293 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:44.132200    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:46.138478 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:46.138438    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:47.545794 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.545757    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg"]
Oct 30 22:45:47.546723 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546699    4889 cpu_manager.go:414] "containerMap: removing container" podUID="50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa" containerName="cleanup"
Oct 30 22:45:47.546723 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546724    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546738    4889 cpu_manager.go:414] "containerMap: removing container" podUID="3d953f06-b11d-4115-8b64-3970e1639126" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546746    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d953f06-b11d-4115-8b64-3970e1639126" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546764    4889 cpu_manager.go:414] "containerMap: removing container" podUID="dfe9d27e-3639-4085-9f3d-93d8a9ec5063" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546771    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfe9d27e-3639-4085-9f3d-93d8a9ec5063" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546784    4889 cpu_manager.go:414] "containerMap: removing container" podUID="ec7b92fc-0bac-45c3-9adb-b0cd06b33198" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546788    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec7b92fc-0bac-45c3-9adb-b0cd06b33198" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546804    4889 cpu_manager.go:414] "containerMap: removing container" podUID="23d07912-2775-4414-9754-d3e16bdf69d2" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546809    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d07912-2775-4414-9754-d3e16bdf69d2" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546816    4889 cpu_manager.go:414] "containerMap: removing container" podUID="23d07912-2775-4414-9754-d3e16bdf69d2" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546820    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d07912-2775-4414-9754-d3e16bdf69d2" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546834    4889 cpu_manager.go:414] "containerMap: removing container" podUID="62aab7d6-3e77-4389-8f85-ce60b5f90056" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546839    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62aab7d6-3e77-4389-8f85-ce60b5f90056" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546847    4889 cpu_manager.go:414] "containerMap: removing container" podUID="62aab7d6-3e77-4389-8f85-ce60b5f90056" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546852    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62aab7d6-3e77-4389-8f85-ce60b5f90056" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546860    4889 cpu_manager.go:414] "containerMap: removing container" podUID="ec7b92fc-0bac-45c3-9adb-b0cd06b33198" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546865    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec7b92fc-0bac-45c3-9adb-b0cd06b33198" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546874    4889 cpu_manager.go:414] "containerMap: removing container" podUID="50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546878    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546885    4889 cpu_manager.go:414] "containerMap: removing container" podUID="3d953f06-b11d-4115-8b64-3970e1639126" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546890    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d953f06-b11d-4115-8b64-3970e1639126" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546897    4889 cpu_manager.go:414] "containerMap: removing container" podUID="b83f41b5-12fa-4cd6-896c-5639ead76511" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546901    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b83f41b5-12fa-4cd6-896c-5639ead76511" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546909    4889 cpu_manager.go:414] "containerMap: removing container" podUID="dfe9d27e-3639-4085-9f3d-93d8a9ec5063" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546913    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfe9d27e-3639-4085-9f3d-93d8a9ec5063" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546922    4889 cpu_manager.go:414] "containerMap: removing container" podUID="3530f8ff-0e20-4062-8939-aa465f065f24" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546927    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3530f8ff-0e20-4062-8939-aa465f065f24" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546935    4889 cpu_manager.go:414] "containerMap: removing container" podUID="3530f8ff-0e20-4062-8939-aa465f065f24" containerName="cleanup"
Oct 30 22:45:47.546934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546940    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3530f8ff-0e20-4062-8939-aa465f065f24" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546951    4889 cpu_manager.go:414] "containerMap: removing container" podUID="78b13226-d659-4865-9627-024fd2ced764" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546956    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78b13226-d659-4865-9627-024fd2ced764" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546966    4889 cpu_manager.go:414] "containerMap: removing container" podUID="78b13226-d659-4865-9627-024fd2ced764" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546970    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78b13226-d659-4865-9627-024fd2ced764" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546977    4889 cpu_manager.go:414] "containerMap: removing container" podUID="78b13226-d659-4865-9627-024fd2ced764" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546981    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78b13226-d659-4865-9627-024fd2ced764" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546989    4889 cpu_manager.go:414] "containerMap: removing container" podUID="3530f8ff-0e20-4062-8939-aa465f065f24" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.546993    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3530f8ff-0e20-4062-8939-aa465f065f24" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.547009    4889 cpu_manager.go:414] "containerMap: removing container" podUID="b83f41b5-12fa-4cd6-896c-5639ead76511" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.547013    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b83f41b5-12fa-4cd6-896c-5639ead76511" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.547163    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="b83f41b5-12fa-4cd6-896c-5639ead76511" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.547172    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="b83f41b5-12fa-4cd6-896c-5639ead76511" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.547177    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d07912-2775-4414-9754-d3e16bdf69d2" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.547186    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.547192    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d953f06-b11d-4115-8b64-3970e1639126" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.547198    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfe9d27e-3639-4085-9f3d-93d8a9ec5063" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.547203    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec7b92fc-0bac-45c3-9adb-b0cd06b33198" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.547210    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.547215    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d07912-2775-4414-9754-d3e16bdf69d2" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.547237    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d07912-2775-4414-9754-d3e16bdf69d2" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.547248    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="62aab7d6-3e77-4389-8f85-ce60b5f90056" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.547256    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="62aab7d6-3e77-4389-8f85-ce60b5f90056" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.547261    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec7b92fc-0bac-45c3-9adb-b0cd06b33198" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.547268    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.547275    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d953f06-b11d-4115-8b64-3970e1639126" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.547281    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfe9d27e-3639-4085-9f3d-93d8a9ec5063" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.547286    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="3530f8ff-0e20-4062-8939-aa465f065f24" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.547292    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="3530f8ff-0e20-4062-8939-aa465f065f24" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.547299    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec7b92fc-0bac-45c3-9adb-b0cd06b33198" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.547305    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="78b13226-d659-4865-9627-024fd2ced764" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.547313    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="78b13226-d659-4865-9627-024fd2ced764" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.547317    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="78b13226-d659-4865-9627-024fd2ced764" containerName="cleanup"
Oct 30 22:45:47.547824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.547322    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="3530f8ff-0e20-4062-8939-aa465f065f24" containerName="cleanup"
Oct 30 22:45:47.553476 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.553452    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg"
Oct 30 22:45:47.570860 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.570826    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg"]
Oct 30 22:45:47.577698 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.577668    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-c511a3-serving-cert\""
Oct 30 22:45:47.667188 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.667144    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg\" (UID: \"bdd7407b-f54b-4e2a-8215-1b1a740b1d69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg"
Oct 30 22:45:47.667433 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.667266    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg\" (UID: \"bdd7407b-f54b-4e2a-8215-1b1a740b1d69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg"
Oct 30 22:45:47.667433 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.667371    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg\" (UID: \"bdd7407b-f54b-4e2a-8215-1b1a740b1d69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg"
Oct 30 22:45:47.667528 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.667501    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgb8l\" (UniqueName: \"kubernetes.io/projected/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-kube-api-access-pgb8l\") pod \"maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg\" (UID: \"bdd7407b-f54b-4e2a-8215-1b1a740b1d69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg"
Oct 30 22:45:47.688113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.688067    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:47.783930 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.783893    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg\" (UID: \"bdd7407b-f54b-4e2a-8215-1b1a740b1d69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg"
Oct 30 22:45:47.784148 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.783981    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg\" (UID: \"bdd7407b-f54b-4e2a-8215-1b1a740b1d69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg"
Oct 30 22:45:47.784148 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.784026    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg\" (UID: \"bdd7407b-f54b-4e2a-8215-1b1a740b1d69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg"
Oct 30 22:45:47.784148 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.784113    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgb8l\" (UniqueName: \"kubernetes.io/projected/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-kube-api-access-pgb8l\") pod \"maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg\" (UID: \"bdd7407b-f54b-4e2a-8215-1b1a740b1d69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg"
Oct 30 22:45:47.784916 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.784891    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg\" (UID: \"bdd7407b-f54b-4e2a-8215-1b1a740b1d69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg"
Oct 30 22:45:47.787158 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.787130    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg\" (UID: \"bdd7407b-f54b-4e2a-8215-1b1a740b1d69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg"
Oct 30 22:45:47.787316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.787295    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg\" (UID: \"bdd7407b-f54b-4e2a-8215-1b1a740b1d69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg"
Oct 30 22:45:47.809034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.808949    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgb8l\" (UniqueName: \"kubernetes.io/projected/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-kube-api-access-pgb8l\") pod \"maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg\" (UID: \"bdd7407b-f54b-4e2a-8215-1b1a740b1d69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg"
Oct 30 22:45:47.870567 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.870530    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:47.872465 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:47.872445    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg"
Oct 30 22:45:48.189869 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:45:48.189825    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd7407b_f54b_4e2a_8215_1b1a740b1d69.slice/crio-d9beba1817f3c1ff96ce2c35adb9892e86aeee1127fd3b1238c0323bbaca001b WatchSource:0}: Error finding container d9beba1817f3c1ff96ce2c35adb9892e86aeee1127fd3b1238c0323bbaca001b: Status 404 returned error can't find the container with id d9beba1817f3c1ff96ce2c35adb9892e86aeee1127fd3b1238c0323bbaca001b
Oct 30 22:45:48.190679 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:48.190648    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg"]
Oct 30 22:45:48.191758 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:48.191737    4889 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:45:48.808927 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:48.808218    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg" event={"ID":"bdd7407b-f54b-4e2a-8215-1b1a740b1d69","Type":"ContainerStarted","Data":"d2b876b0153e506aff6408c123693f5d45dbc5fcb04e3ce848d2cbc3ec69f0ca"}
Oct 30 22:45:48.808927 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:48.808349    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg" event={"ID":"bdd7407b-f54b-4e2a-8215-1b1a740b1d69","Type":"ContainerStarted","Data":"d9beba1817f3c1ff96ce2c35adb9892e86aeee1127fd3b1238c0323bbaca001b"}
Oct 30 22:45:48.808927 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:48.808408    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg"
Oct 30 22:45:48.856027 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:48.855968    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg" podStartSLOduration=1.541355324 podStartE2EDuration="1.855950286s" totalImagesPullingTime="314.594962ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:48.850627161 +0000 UTC m=+2334.206993477" watchObservedRunningTime="2026-10-30 22:45:48.855950286 +0000 UTC m=+2334.212316600"
Oct 30 22:45:49.415248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:49.414813    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:49.576675 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:49.576623    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:51.173937 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:51.173901    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:52.900404 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:52.900366    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:54.621408 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:54.621367    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:54.823582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:54.823553    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg"
Oct 30 22:45:56.419999 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:56.419958    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:58.156700 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:58.155510    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:59.899978 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:45:59.899933    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:00.702878 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:00.702829    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:01.711609 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:01.711553    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:03.428856 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:03.428813    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:05.353496 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:05.353447    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:07.019799 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:07.019767    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:08.710641 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:08.710595    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:10.304690 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:46:10.304652    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:46:10.445488 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:10.445449    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:12.131724 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:12.131661    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:13.815749 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:13.815708    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:15.531246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:15.528526    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:17.228003 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:17.227960    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:18.890619 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:18.890582    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:19.257781 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:19.257740    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"]
Oct 30 22:46:19.346941 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:19.346873    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:20.954356 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:20.954313    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:22.797310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:22.774472    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:24.539084 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:24.539041    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:26.397624 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:26.397577    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"]
Oct 30 22:46:26.398198 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:26.397791    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c" podUID="78af4d09-3007-46d3-9b8e-043a4f536c26" containerName="istio-proxy" containerID="cri-o://fd40f2adc8766c131ef7b56112904978b9e06f409d0205cda5e3f88b2cf4cc29" gracePeriod=30
Oct 30 22:46:26.437280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:26.432895    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:28.346292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:28.336936    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:30.072391 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:30.072352    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:30.518488 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:30.518450    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:30.700442 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:30.700399    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:31.722255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.722205    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:46:31.809037 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.808989    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-data\") pod \"78af4d09-3007-46d3-9b8e-043a4f536c26\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") "
Oct 30 22:46:31.809311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.809075    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-token\" (UniqueName: \"kubernetes.io/projected/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-token\") pod \"78af4d09-3007-46d3-9b8e-043a4f536c26\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") "
Oct 30 22:46:31.809413 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.809309    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-workload-socket\") pod \"78af4d09-3007-46d3-9b8e-043a4f536c26\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") "
Oct 30 22:46:31.809413 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.809372    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-envoy\") pod \"78af4d09-3007-46d3-9b8e-043a4f536c26\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") "
Oct 30 22:46:31.809548 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.809418    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-data" pod "78af4d09-3007-46d3-9b8e-043a4f536c26" (UID: "78af4d09-3007-46d3-9b8e-043a4f536c26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:31.809548 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.809478    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-credential-socket\") pod \"78af4d09-3007-46d3-9b8e-043a4f536c26\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") "
Oct 30 22:46:31.809666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.809589    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-workload-certs\") pod \"78af4d09-3007-46d3-9b8e-043a4f536c26\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") "
Oct 30 22:46:31.809666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.809652    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78af4d09-3007-46d3-9b8e-043a4f536c26-kube-api-access-mnhgn\" (UniqueName: \"kubernetes.io/projected/78af4d09-3007-46d3-9b8e-043a4f536c26-kube-api-access-mnhgn\") pod \"78af4d09-3007-46d3-9b8e-043a4f536c26\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") "
Oct 30 22:46:31.809783 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.809681    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-podinfo\") pod \"78af4d09-3007-46d3-9b8e-043a4f536c26\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") "
Oct 30 22:46:31.809783 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.809741    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78af4d09-3007-46d3-9b8e-043a4f536c26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78af4d09-3007-46d3-9b8e-043a4f536c26-istiod-ca-cert\") pod \"78af4d09-3007-46d3-9b8e-043a4f536c26\" (UID: \"78af4d09-3007-46d3-9b8e-043a4f536c26\") "
Oct 30 22:46:31.810712 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.810039    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-credential-socket" pod "78af4d09-3007-46d3-9b8e-043a4f536c26" (UID: "78af4d09-3007-46d3-9b8e-043a4f536c26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:31.810712 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.810424    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78af4d09-3007-46d3-9b8e-043a4f536c26-istiod-ca-cert" pod "78af4d09-3007-46d3-9b8e-043a4f536c26" (UID: "78af4d09-3007-46d3-9b8e-043a4f536c26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:31.810712 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.810624    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:46:31.810712 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.810642    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78af4d09-3007-46d3-9b8e-043a4f536c26-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:46:31.810712 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.810658    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:46:31.811416 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.811217    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-workload-socket" pod "78af4d09-3007-46d3-9b8e-043a4f536c26" (UID: "78af4d09-3007-46d3-9b8e-043a4f536c26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:31.811537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.811286    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-workload-certs" pod "78af4d09-3007-46d3-9b8e-043a4f536c26" (UID: "78af4d09-3007-46d3-9b8e-043a4f536c26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:31.814350 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.814306    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78af4d09-3007-46d3-9b8e-043a4f536c26-kube-api-access-mnhgn" pod "78af4d09-3007-46d3-9b8e-043a4f536c26" (UID: "78af4d09-3007-46d3-9b8e-043a4f536c26"). InnerVolumeSpecName "kube-api-access-mnhgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:31.815784 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.815744    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-token" pod "78af4d09-3007-46d3-9b8e-043a4f536c26" (UID: "78af4d09-3007-46d3-9b8e-043a4f536c26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:31.816389 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.816358    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-podinfo" pod "78af4d09-3007-46d3-9b8e-043a4f536c26" (UID: "78af4d09-3007-46d3-9b8e-043a4f536c26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:46:31.816825 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.816800    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-envoy" pod "78af4d09-3007-46d3-9b8e-043a4f536c26" (UID: "78af4d09-3007-46d3-9b8e-043a4f536c26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:31.912498 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.912464    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:46:31.912498 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.912496    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:46:31.912498 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.912506    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:46:31.912773 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.912516    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78af4d09-3007-46d3-9b8e-043a4f536c26-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:46:31.912773 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.912526    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnhgn\" (UniqueName: \"kubernetes.io/projected/78af4d09-3007-46d3-9b8e-043a4f536c26-kube-api-access-mnhgn\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:46:31.912773 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.912534    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78af4d09-3007-46d3-9b8e-043a4f536c26-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:46:31.944966 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:31.944784    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:32.212058 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:32.211969    4889 generic.go:356] "Generic (PLEG): container finished" podID="78af4d09-3007-46d3-9b8e-043a4f536c26" containerID="fd40f2adc8766c131ef7b56112904978b9e06f409d0205cda5e3f88b2cf4cc29" exitCode=0
Oct 30 22:46:32.212058 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:32.212043    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"
Oct 30 22:46:32.212323 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:32.212066    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c" event={"ID":"78af4d09-3007-46d3-9b8e-043a4f536c26","Type":"ContainerDied","Data":"fd40f2adc8766c131ef7b56112904978b9e06f409d0205cda5e3f88b2cf4cc29"}
Oct 30 22:46:32.212323 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:32.212126    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c" event={"ID":"78af4d09-3007-46d3-9b8e-043a4f536c26","Type":"ContainerDied","Data":"3fb148465e6906fee6236f65c5324fbc74dda8b16f53b231d1515f3b0748bb33"}
Oct 30 22:46:32.212323 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:32.212154    4889 scope.go:122] "RemoveContainer" containerID="fd40f2adc8766c131ef7b56112904978b9e06f409d0205cda5e3f88b2cf4cc29"
Oct 30 22:46:32.231644 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:32.231620    4889 scope.go:122] "RemoveContainer" containerID="fd40f2adc8766c131ef7b56112904978b9e06f409d0205cda5e3f88b2cf4cc29"
Oct 30 22:46:32.232060 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:46:32.232019    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd40f2adc8766c131ef7b56112904978b9e06f409d0205cda5e3f88b2cf4cc29\": container with ID starting with fd40f2adc8766c131ef7b56112904978b9e06f409d0205cda5e3f88b2cf4cc29 not found: ID does not exist" containerID="fd40f2adc8766c131ef7b56112904978b9e06f409d0205cda5e3f88b2cf4cc29"
Oct 30 22:46:32.232172 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:32.232062    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd40f2adc8766c131ef7b56112904978b9e06f409d0205cda5e3f88b2cf4cc29"} err="failed to get container status \"fd40f2adc8766c131ef7b56112904978b9e06f409d0205cda5e3f88b2cf4cc29\": rpc error: code = NotFound desc = could not find container \"fd40f2adc8766c131ef7b56112904978b9e06f409d0205cda5e3f88b2cf4cc29\": container with ID starting with fd40f2adc8766c131ef7b56112904978b9e06f409d0205cda5e3f88b2cf4cc29 not found: ID does not exist"
Oct 30 22:46:32.274083 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:32.274048    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"]
Oct 30 22:46:32.281377 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:32.281342    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-t6h8c"]
Oct 30 22:46:32.778057 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:32.778017    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78af4d09-3007-46d3-9b8e-043a4f536c26" path="/var/lib/kubelet/pods/78af4d09-3007-46d3-9b8e-043a4f536c26/volumes"
Oct 30 22:46:33.774098 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:33.769123    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:34.237546 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:34.237505    4889 generic.go:356] "Generic (PLEG): container finished" podID="5a707459-e53d-486b-a1e5-0456ab08570c" containerID="981bd4e0cb5bd05d6b76026ccab69712b1d990a628fddfae3a194e1f3c7d624c" exitCode=0
Oct 30 22:46:34.238071 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:34.237562    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr" event={"ID":"5a707459-e53d-486b-a1e5-0456ab08570c","Type":"ContainerDied","Data":"981bd4e0cb5bd05d6b76026ccab69712b1d990a628fddfae3a194e1f3c7d624c"}
Oct 30 22:46:34.238183 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:34.238163    4889 scope.go:122] "RemoveContainer" containerID="981bd4e0cb5bd05d6b76026ccab69712b1d990a628fddfae3a194e1f3c7d624c"
Oct 30 22:46:34.240555 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:34.240527    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:46:34.240699 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:34.240620    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:46:34.240699 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:34.240689    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:46:34.287829 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:34.287794    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t" podUID="0b4e3337-ba72-4247-b521-85740c5a5c3a" containerName="main" containerID="cri-o://17c6c74500939f16ab15280c9df6a86374198bfbce877381cb4f1268cb22a832" gracePeriod=45
Oct 30 22:46:34.454285 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:34.452867    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t" podUID="0b4e3337-ba72-4247-b521-85740c5a5c3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.179:8000/ready\": dial tcp 10.132.0.179:8000: connect: connection refused"
Oct 30 22:46:34.649897 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:34.649858    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"
Oct 30 22:46:34.767721 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:34.767630    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-tmp-dir\") pod \"0b4e3337-ba72-4247-b521-85740c5a5c3a\" (UID: \"0b4e3337-ba72-4247-b521-85740c5a5c3a\") "
Oct 30 22:46:34.767721 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:34.767673    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-home\" (UniqueName: \"kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-home\") pod \"0b4e3337-ba72-4247-b521-85740c5a5c3a\" (UID: \"0b4e3337-ba72-4247-b521-85740c5a5c3a\") "
Oct 30 22:46:34.767721 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:34.767706    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-dshm\") pod \"0b4e3337-ba72-4247-b521-85740c5a5c3a\" (UID: \"0b4e3337-ba72-4247-b521-85740c5a5c3a\") "
Oct 30 22:46:34.768017 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:34.767932    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-tmp-dir" pod "0b4e3337-ba72-4247-b521-85740c5a5c3a" (UID: "0b4e3337-ba72-4247-b521-85740c5a5c3a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:34.768080 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:34.768022    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b4e3337-ba72-4247-b521-85740c5a5c3a-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b4e3337-ba72-4247-b521-85740c5a5c3a-tls-certs\") pod \"0b4e3337-ba72-4247-b521-85740c5a5c3a\" (UID: \"0b4e3337-ba72-4247-b521-85740c5a5c3a\") "
Oct 30 22:46:34.768080 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:34.768025    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-home" pod "0b4e3337-ba72-4247-b521-85740c5a5c3a" (UID: "0b4e3337-ba72-4247-b521-85740c5a5c3a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:34.768196 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:34.768086    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-model-cache\") pod \"0b4e3337-ba72-4247-b521-85740c5a5c3a\" (UID: \"0b4e3337-ba72-4247-b521-85740c5a5c3a\") "
Oct 30 22:46:34.768196 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:34.768162    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b4e3337-ba72-4247-b521-85740c5a5c3a-kube-api-access-8flst\" (UniqueName: \"kubernetes.io/projected/0b4e3337-ba72-4247-b521-85740c5a5c3a-kube-api-access-8flst\") pod \"0b4e3337-ba72-4247-b521-85740c5a5c3a\" (UID: \"0b4e3337-ba72-4247-b521-85740c5a5c3a\") "
Oct 30 22:46:34.768541 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:34.768432    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-model-cache" pod "0b4e3337-ba72-4247-b521-85740c5a5c3a" (UID: "0b4e3337-ba72-4247-b521-85740c5a5c3a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:34.769017 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:34.768951    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:46:34.769017 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:34.768985    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:46:34.769017 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:34.768998    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:46:34.770956 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:34.770928    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b4e3337-ba72-4247-b521-85740c5a5c3a-kube-api-access-8flst" pod "0b4e3337-ba72-4247-b521-85740c5a5c3a" (UID: "0b4e3337-ba72-4247-b521-85740c5a5c3a"). InnerVolumeSpecName "kube-api-access-8flst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:34.771077 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:34.770978    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b4e3337-ba72-4247-b521-85740c5a5c3a-tls-certs" pod "0b4e3337-ba72-4247-b521-85740c5a5c3a" (UID: "0b4e3337-ba72-4247-b521-85740c5a5c3a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:46:34.771077 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:34.771036    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-dshm" pod "0b4e3337-ba72-4247-b521-85740c5a5c3a" (UID: "0b4e3337-ba72-4247-b521-85740c5a5c3a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:34.871503 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:34.871466    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b4e3337-ba72-4247-b521-85740c5a5c3a-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:46:34.871503 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:34.871506    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8flst\" (UniqueName: \"kubernetes.io/projected/0b4e3337-ba72-4247-b521-85740c5a5c3a-kube-api-access-8flst\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:46:34.871744 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:34.871521    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b4e3337-ba72-4247-b521-85740c5a5c3a-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:46:35.254779 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:35.254746    4889 generic.go:356] "Generic (PLEG): container finished" podID="0b4e3337-ba72-4247-b521-85740c5a5c3a" containerID="17c6c74500939f16ab15280c9df6a86374198bfbce877381cb4f1268cb22a832" exitCode=0
Oct 30 22:46:35.255320 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:35.254806    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"
Oct 30 22:46:35.255320 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:35.254827    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t" event={"ID":"0b4e3337-ba72-4247-b521-85740c5a5c3a","Type":"ContainerDied","Data":"17c6c74500939f16ab15280c9df6a86374198bfbce877381cb4f1268cb22a832"}
Oct 30 22:46:35.255320 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:35.254875    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t" event={"ID":"0b4e3337-ba72-4247-b521-85740c5a5c3a","Type":"ContainerDied","Data":"12b8bafa30aefb55dbd134d74abd5a0579d011b05493740828528187e2d3e0b3"}
Oct 30 22:46:35.255320 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:35.254900    4889 scope.go:122] "RemoveContainer" containerID="17c6c74500939f16ab15280c9df6a86374198bfbce877381cb4f1268cb22a832"
Oct 30 22:46:35.257981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:35.257931    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr" event={"ID":"5a707459-e53d-486b-a1e5-0456ab08570c","Type":"ContainerStarted","Data":"6affa35bb38bd78180c641c0599cf7c658d43284dd518ebf04357db101246737"}
Oct 30 22:46:35.266917 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:35.266895    4889 scope.go:122] "RemoveContainer" containerID="17c6c74500939f16ab15280c9df6a86374198bfbce877381cb4f1268cb22a832"
Oct 30 22:46:35.267298 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:46:35.267276    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17c6c74500939f16ab15280c9df6a86374198bfbce877381cb4f1268cb22a832\": container with ID starting with 17c6c74500939f16ab15280c9df6a86374198bfbce877381cb4f1268cb22a832 not found: ID does not exist" containerID="17c6c74500939f16ab15280c9df6a86374198bfbce877381cb4f1268cb22a832"
Oct 30 22:46:35.267377 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:35.267308    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17c6c74500939f16ab15280c9df6a86374198bfbce877381cb4f1268cb22a832"} err="failed to get container status \"17c6c74500939f16ab15280c9df6a86374198bfbce877381cb4f1268cb22a832\": rpc error: code = NotFound desc = could not find container \"17c6c74500939f16ab15280c9df6a86374198bfbce877381cb4f1268cb22a832\": container with ID starting with 17c6c74500939f16ab15280c9df6a86374198bfbce877381cb4f1268cb22a832 not found: ID does not exist"
Oct 30 22:46:35.375189 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:35.375153    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"]
Oct 30 22:46:35.380304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:35.380271    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-54c7f0/test-model-54c7f0-kserve-94c596b77-bjl5t"]
Oct 30 22:46:35.531312 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:35.526917    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:35.908235 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:35.908118    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:46:35.908493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:35.908472    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:46:35.908549 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:35.908518    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr" podUID="5a707459-e53d-486b-a1e5-0456ab08570c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 22:46:36.788095 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:36.788035    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b4e3337-ba72-4247-b521-85740c5a5c3a" path="/var/lib/kubelet/pods/0b4e3337-ba72-4247-b521-85740c5a5c3a/volumes"
Oct 30 22:46:36.908079 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:36.908041    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:46:36.908331 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:36.908102    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr" podUID="5a707459-e53d-486b-a1e5-0456ab08570c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 22:46:37.425437 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:37.425348    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:37.908735 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:37.908698    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:46:37.909148 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:37.908774    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr" podUID="5a707459-e53d-486b-a1e5-0456ab08570c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 22:46:38.908877 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:38.908840    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:46:38.909350 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:38.908896    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr" podUID="5a707459-e53d-486b-a1e5-0456ab08570c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 22:46:39.444601 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:39.444562    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:39.907895 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:39.907855    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:46:39.908107 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:39.907917    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr" podUID="5a707459-e53d-486b-a1e5-0456ab08570c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 22:46:40.306803 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:46:40.306760    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:46:41.145584 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:41.145545    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:46:41.146096 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:41.146066    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:46:41.147245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:41.147200    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:46:41.250708 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:41.250671    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:42.833488 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:42.833446    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:43.004551 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.004516    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:43.269466 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.269429    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:43.628173 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.628084    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"]
Oct 30 22:46:43.629351 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.629313    4889 cpu_manager.go:414] "containerMap: removing container" podUID="23d07912-2775-4414-9754-d3e16bdf69d2" containerName="cleanup"
Oct 30 22:46:43.629351 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.629341    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d07912-2775-4414-9754-d3e16bdf69d2" containerName="cleanup"
Oct 30 22:46:43.629461 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.629373    4889 cpu_manager.go:414] "containerMap: removing container" podUID="62aab7d6-3e77-4389-8f85-ce60b5f90056" containerName="cleanup"
Oct 30 22:46:43.629461 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.629380    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62aab7d6-3e77-4389-8f85-ce60b5f90056" containerName="cleanup"
Oct 30 22:46:43.629461 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.629397    4889 cpu_manager.go:414] "containerMap: removing container" podUID="ec7b92fc-0bac-45c3-9adb-b0cd06b33198" containerName="cleanup"
Oct 30 22:46:43.629461 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.629406    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec7b92fc-0bac-45c3-9adb-b0cd06b33198" containerName="cleanup"
Oct 30 22:46:43.629461 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.629419    4889 cpu_manager.go:414] "containerMap: removing container" podUID="3d953f06-b11d-4115-8b64-3970e1639126" containerName="cleanup"
Oct 30 22:46:43.629461 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.629426    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d953f06-b11d-4115-8b64-3970e1639126" containerName="cleanup"
Oct 30 22:46:43.629461 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.629437    4889 cpu_manager.go:414] "containerMap: removing container" podUID="b83f41b5-12fa-4cd6-896c-5639ead76511" containerName="cleanup"
Oct 30 22:46:43.629461 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.629444    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b83f41b5-12fa-4cd6-896c-5639ead76511" containerName="cleanup"
Oct 30 22:46:43.629726 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.629489    4889 cpu_manager.go:414] "containerMap: removing container" podUID="0b4e3337-ba72-4247-b521-85740c5a5c3a" containerName="main"
Oct 30 22:46:43.629726 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.629496    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b4e3337-ba72-4247-b521-85740c5a5c3a" containerName="main"
Oct 30 22:46:43.629726 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.629533    4889 cpu_manager.go:414] "containerMap: removing container" podUID="50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa" containerName="cleanup"
Oct 30 22:46:43.629726 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.629539    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50ccda0b-60a9-4d7c-a265-9fdeb90bcdaa" containerName="cleanup"
Oct 30 22:46:43.629726 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.629555    4889 cpu_manager.go:414] "containerMap: removing container" podUID="dfe9d27e-3639-4085-9f3d-93d8a9ec5063" containerName="cleanup"
Oct 30 22:46:43.629726 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.629561    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfe9d27e-3639-4085-9f3d-93d8a9ec5063" containerName="cleanup"
Oct 30 22:46:43.629726 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.629582    4889 cpu_manager.go:414] "containerMap: removing container" podUID="78af4d09-3007-46d3-9b8e-043a4f536c26" containerName="istio-proxy"
Oct 30 22:46:43.629726 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.629589    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78af4d09-3007-46d3-9b8e-043a4f536c26" containerName="istio-proxy"
Oct 30 22:46:43.629982 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.629770    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b4e3337-ba72-4247-b521-85740c5a5c3a" containerName="main"
Oct 30 22:46:43.629982 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.629784    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="b83f41b5-12fa-4cd6-896c-5639ead76511" containerName="cleanup"
Oct 30 22:46:43.629982 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.629796    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d953f06-b11d-4115-8b64-3970e1639126" containerName="cleanup"
Oct 30 22:46:43.629982 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.629803    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfe9d27e-3639-4085-9f3d-93d8a9ec5063" containerName="cleanup"
Oct 30 22:46:43.629982 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.629811    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="78af4d09-3007-46d3-9b8e-043a4f536c26" containerName="istio-proxy"
Oct 30 22:46:43.629982 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.629821    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="62aab7d6-3e77-4389-8f85-ce60b5f90056" containerName="cleanup"
Oct 30 22:46:43.636105 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.636075    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.658936 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.658897    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"]
Oct 30 22:46:43.662031 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.661997    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-b03b33-openshift-default-dockercfg-tb8rk\""
Oct 30 22:46:43.762254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.745742    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-podinfo\") pod \"e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") " pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.762254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.745815    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-envoy\") pod \"e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") " pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.762254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.745870    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istiod-ca-cert\") pod \"e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") " pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.762254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.745920    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-token\") pod \"e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") " pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.762254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.746043    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-data\") pod \"e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") " pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.762254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.746074    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-credential-socket\") pod \"e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") " pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.762254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.746215    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pglfg\" (UniqueName: \"kubernetes.io/projected/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-kube-api-access-pglfg\") pod \"e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") " pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.762254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.746324    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-workload-socket\") pod \"e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") " pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.762254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.746365    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-workload-certs\") pod \"e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") " pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.762254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.755498    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:43.857445 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.857405    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-data\") pod \"e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") " pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.858086 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.858056    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-credential-socket\") pod \"e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") " pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.858371 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.858349    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pglfg\" (UniqueName: \"kubernetes.io/projected/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-kube-api-access-pglfg\") pod \"e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") " pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.858586 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.858569    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-workload-socket\") pod \"e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") " pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.858716 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.858700    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-workload-certs\") pod \"e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") " pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.858891 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.858877    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-podinfo\") pod \"e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") " pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.859037 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.859020    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-envoy\") pod \"e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") " pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.859188 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.859171    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istiod-ca-cert\") pod \"e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") " pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.859401 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.859381    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-token\") pod \"e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") " pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.860368 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.860344    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-data\") pod \"e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") " pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.868239 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.861077    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-credential-socket\") pod \"e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") " pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.868516 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.861684    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-workload-socket\") pod \"e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") " pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.868696 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.861896    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-workload-certs\") pod \"e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") " pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.868836 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.862827    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istiod-ca-cert\") pod \"e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") " pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.873351 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.873310    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-podinfo\") pod \"e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") " pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.887118 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.887000    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-token\") pod \"e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") " pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.888250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.888157    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pglfg\" (UniqueName: \"kubernetes.io/projected/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-kube-api-access-pglfg\") pod \"e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") " pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.891323 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.891155    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-envoy\") pod \"e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") " pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:43.971837 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:43.971799    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:44.214541 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:44.214437    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:44.325920 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:46:44.325884    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb5b6d8_8552_46dc_9ef3_26f8fd64a8b9.slice/crio-de97f411d1fcf832865b082d40a5f5091ac66b1c3e5bbceee6db36e7b800692a WatchSource:0}: Error finding container de97f411d1fcf832865b082d40a5f5091ac66b1c3e5bbceee6db36e7b800692a: Status 404 returned error can't find the container with id de97f411d1fcf832865b082d40a5f5091ac66b1c3e5bbceee6db36e7b800692a
Oct 30 22:46:44.328912 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:44.328881    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"]
Oct 30 22:46:44.329336 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:44.329304    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:46:44.329424 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:44.329375    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:46:44.329424 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:44.329407    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:46:44.350808 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:44.350771    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd" event={"ID":"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9","Type":"ContainerStarted","Data":"de97f411d1fcf832865b082d40a5f5091ac66b1c3e5bbceee6db36e7b800692a"}
Oct 30 22:46:44.716588 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:44.716545    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:45.361010 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:45.360969    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd" event={"ID":"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9","Type":"ContainerStarted","Data":"472db4676bd2b485e4f867a6a49d2aad2e7babee5603cd02514bcfefb1a8bee6"}
Oct 30 22:46:45.422018 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:45.421966    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd" podStartSLOduration=2.42195369 podStartE2EDuration="2.42195369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:45.42074072 +0000 UTC m=+2390.777107032" watchObservedRunningTime="2026-10-30 22:46:45.42195369 +0000 UTC m=+2390.778320004"
Oct 30 22:46:45.972470 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:45.972431    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:45.979506 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:45.979473    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:46.370103 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:46.370066    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:46.371288 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:46.371211    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:46:46.416769 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:46.416706    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:47.909759 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:47.909714    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:48.081747 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:48.081716    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:48.360919 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:48.360885    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:49.573357 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:49.573316    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:49.747151 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:49.747102    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:49.915540 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:49.915402    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:50.256582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:50.256546    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:51.966635 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:51.966487    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:53.754007 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:53.753959    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:55.507200 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:55.507146    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:55.614559 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:55.614529    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 22:46:55.616827 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:55.616799    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 22:46:57.261552 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:57.261495    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:59.229536 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:59.229492    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:59.787807 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:46:59.787769    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:00.157409 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:00.157313    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"]
Oct 30 22:47:00.327251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:00.313911    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:01.126959 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:01.126916    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:02.836646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:02.836589    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:04.870956 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:04.870916    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:06.305997 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:06.305959    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:07.781377 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:07.781339    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:08.385347 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:08.385256    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:08.547983 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:08.547945    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:08.713849 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:08.713743    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:09.061883 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:09.061848    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:10.307486 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:47:10.307444    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:47:10.635679 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:10.635586    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:12.279246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:12.270626    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:14.043970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:14.043935    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:15.188877 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.188832    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w" podUID="3b0d1da0-5e38-4000-826a-01b8110db1a1" containerName="main" containerID="cri-o://7390b7477a2a96d2e10b883e9e674df80f6a2290f46babd82660be5f9fb93edc" gracePeriod=45
Oct 30 22:47:15.292009 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.291930    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:15.458770 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.458682    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:15.532837 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.532806    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"
Oct 30 22:47:15.546138 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.546094    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"]
Oct 30 22:47:15.546424 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.546390    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd" podUID="b0758d58-495a-4f8e-a1f6-daaca5024f8c" containerName="istio-proxy" containerID="cri-o://ffc6f8ccb1ed86992599d55bbbe3f5a0caa42bcd4fda7cd67357c624a2bb9209" gracePeriod=30
Oct 30 22:47:15.628927 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.628873    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0d1da0-5e38-4000-826a-01b8110db1a1-kube-api-access-cn5bv\" (UniqueName: \"kubernetes.io/projected/3b0d1da0-5e38-4000-826a-01b8110db1a1-kube-api-access-cn5bv\") pod \"3b0d1da0-5e38-4000-826a-01b8110db1a1\" (UID: \"3b0d1da0-5e38-4000-826a-01b8110db1a1\") "
Oct 30 22:47:15.629145 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.628962    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-home\" (UniqueName: \"kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-home\") pod \"3b0d1da0-5e38-4000-826a-01b8110db1a1\" (UID: \"3b0d1da0-5e38-4000-826a-01b8110db1a1\") "
Oct 30 22:47:15.629145 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.629126    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-dshm\") pod \"3b0d1da0-5e38-4000-826a-01b8110db1a1\" (UID: \"3b0d1da0-5e38-4000-826a-01b8110db1a1\") "
Oct 30 22:47:15.629313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.629160    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b0d1da0-5e38-4000-826a-01b8110db1a1-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b0d1da0-5e38-4000-826a-01b8110db1a1-tls-certs\") pod \"3b0d1da0-5e38-4000-826a-01b8110db1a1\" (UID: \"3b0d1da0-5e38-4000-826a-01b8110db1a1\") "
Oct 30 22:47:15.629387 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.629300    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-home" pod "3b0d1da0-5e38-4000-826a-01b8110db1a1" (UID: "3b0d1da0-5e38-4000-826a-01b8110db1a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:15.629387 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.629360    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-model-cache\") pod \"3b0d1da0-5e38-4000-826a-01b8110db1a1\" (UID: \"3b0d1da0-5e38-4000-826a-01b8110db1a1\") "
Oct 30 22:47:15.629698 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.629635    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-tmp-dir\") pod \"3b0d1da0-5e38-4000-826a-01b8110db1a1\" (UID: \"3b0d1da0-5e38-4000-826a-01b8110db1a1\") "
Oct 30 22:47:15.629698 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.629687    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-model-cache" pod "3b0d1da0-5e38-4000-826a-01b8110db1a1" (UID: "3b0d1da0-5e38-4000-826a-01b8110db1a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:15.629924 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.629901    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-tmp-dir" pod "3b0d1da0-5e38-4000-826a-01b8110db1a1" (UID: "3b0d1da0-5e38-4000-826a-01b8110db1a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:15.630769 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.630707    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:15.630769 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.630731    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:15.630769 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.630745    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:15.631310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.631217    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:15.632514 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.632486    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-dshm" pod "3b0d1da0-5e38-4000-826a-01b8110db1a1" (UID: "3b0d1da0-5e38-4000-826a-01b8110db1a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:15.632514 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.632502    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0d1da0-5e38-4000-826a-01b8110db1a1-kube-api-access-cn5bv" pod "3b0d1da0-5e38-4000-826a-01b8110db1a1" (UID: "3b0d1da0-5e38-4000-826a-01b8110db1a1"). InnerVolumeSpecName "kube-api-access-cn5bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:15.632707 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.632593    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b0d1da0-5e38-4000-826a-01b8110db1a1-tls-certs" pod "3b0d1da0-5e38-4000-826a-01b8110db1a1" (UID: "3b0d1da0-5e38-4000-826a-01b8110db1a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:15.710792 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.710705    4889 generic.go:356] "Generic (PLEG): container finished" podID="3b0d1da0-5e38-4000-826a-01b8110db1a1" containerID="7390b7477a2a96d2e10b883e9e674df80f6a2290f46babd82660be5f9fb93edc" exitCode=0
Oct 30 22:47:15.710792 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.710760    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"
Oct 30 22:47:15.710992 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.710787    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w" event={"ID":"3b0d1da0-5e38-4000-826a-01b8110db1a1","Type":"ContainerDied","Data":"7390b7477a2a96d2e10b883e9e674df80f6a2290f46babd82660be5f9fb93edc"}
Oct 30 22:47:15.710992 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.710837    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w" event={"ID":"3b0d1da0-5e38-4000-826a-01b8110db1a1","Type":"ContainerDied","Data":"28cd27f98459d9df05e33ed63a1f81760d8326ddcc9407e859a055f7da6e6755"}
Oct 30 22:47:15.710992 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.710856    4889 scope.go:122] "RemoveContainer" containerID="7390b7477a2a96d2e10b883e9e674df80f6a2290f46babd82660be5f9fb93edc"
Oct 30 22:47:15.724332 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.724303    4889 scope.go:122] "RemoveContainer" containerID="7390b7477a2a96d2e10b883e9e674df80f6a2290f46babd82660be5f9fb93edc"
Oct 30 22:47:15.725134 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:47:15.725053    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7390b7477a2a96d2e10b883e9e674df80f6a2290f46babd82660be5f9fb93edc\": container with ID starting with 7390b7477a2a96d2e10b883e9e674df80f6a2290f46babd82660be5f9fb93edc not found: ID does not exist" containerID="7390b7477a2a96d2e10b883e9e674df80f6a2290f46babd82660be5f9fb93edc"
Oct 30 22:47:15.725134 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.725105    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7390b7477a2a96d2e10b883e9e674df80f6a2290f46babd82660be5f9fb93edc"} err="failed to get container status \"7390b7477a2a96d2e10b883e9e674df80f6a2290f46babd82660be5f9fb93edc\": rpc error: code = NotFound desc = could not find container \"7390b7477a2a96d2e10b883e9e674df80f6a2290f46babd82660be5f9fb93edc\": container with ID starting with 7390b7477a2a96d2e10b883e9e674df80f6a2290f46babd82660be5f9fb93edc not found: ID does not exist"
Oct 30 22:47:15.801274 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.800756    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn5bv\" (UniqueName: \"kubernetes.io/projected/3b0d1da0-5e38-4000-826a-01b8110db1a1-kube-api-access-cn5bv\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:15.801274 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.800796    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b0d1da0-5e38-4000-826a-01b8110db1a1-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:15.801274 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.800812    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b0d1da0-5e38-4000-826a-01b8110db1a1-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:15.865284 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.862644    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:15.869164 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.869106    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"]
Oct 30 22:47:15.877113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:15.876997    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w"]
Oct 30 22:47:16.519664 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:16.519617    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d85da4/test-model-d85da4-kserve-64556965c9-4zh6w" podUID="3b0d1da0-5e38-4000-826a-01b8110db1a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.180:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:47:16.780439 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:16.780356    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b0d1da0-5e38-4000-826a-01b8110db1a1" path="/var/lib/kubelet/pods/3b0d1da0-5e38-4000-826a-01b8110db1a1/volumes"
Oct 30 22:47:17.685607 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:17.685495    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:19.439789 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:19.439720    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:20.065352 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:20.065306    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:20.769887 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:20.769848    4889 generic.go:356] "Generic (PLEG): container finished" podID="b0758d58-495a-4f8e-a1f6-daaca5024f8c" containerID="ffc6f8ccb1ed86992599d55bbbe3f5a0caa42bcd4fda7cd67357c624a2bb9209" exitCode=0
Oct 30 22:47:20.779277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:20.779243    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd" event={"ID":"b0758d58-495a-4f8e-a1f6-daaca5024f8c","Type":"ContainerDied","Data":"ffc6f8ccb1ed86992599d55bbbe3f5a0caa42bcd4fda7cd67357c624a2bb9209"}
Oct 30 22:47:20.779582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:20.779563    4889 scope.go:122] "RemoveContainer" containerID="1441a09a6ac5bf01a605cb2ea241faee3b13fc3c8da4ae0cdb0c0ba560b991b6"
Oct 30 22:47:20.861713 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:20.861685    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:47:20.983907 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:20.983870    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-data\") pod \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") "
Oct 30 22:47:20.983907 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:20.983923    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-envoy\") pod \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") "
Oct 30 22:47:20.984210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:20.984017    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0758d58-495a-4f8e-a1f6-daaca5024f8c-kube-api-access-72r26\" (UniqueName: \"kubernetes.io/projected/b0758d58-495a-4f8e-a1f6-daaca5024f8c-kube-api-access-72r26\") pod \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") "
Oct 30 22:47:20.984210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:20.984115    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-podinfo\") pod \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") "
Oct 30 22:47:20.984210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:20.984172    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istiod-ca-cert\") pod \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") "
Oct 30 22:47:20.984447 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:20.984209    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-workload-certs\") pod \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") "
Oct 30 22:47:20.984447 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:20.984280    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-data" pod "b0758d58-495a-4f8e-a1f6-daaca5024f8c" (UID: "b0758d58-495a-4f8e-a1f6-daaca5024f8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:20.984447 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:20.984326    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-token\" (UniqueName: \"kubernetes.io/projected/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-token\") pod \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") "
Oct 30 22:47:20.984447 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:20.984359    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-credential-socket\") pod \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") "
Oct 30 22:47:20.984683 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:20.984489    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-workload-socket\") pod \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\" (UID: \"b0758d58-495a-4f8e-a1f6-daaca5024f8c\") "
Oct 30 22:47:20.984683 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:20.984625    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-workload-certs" pod "b0758d58-495a-4f8e-a1f6-daaca5024f8c" (UID: "b0758d58-495a-4f8e-a1f6-daaca5024f8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:20.984845 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:20.984819    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istiod-ca-cert" pod "b0758d58-495a-4f8e-a1f6-daaca5024f8c" (UID: "b0758d58-495a-4f8e-a1f6-daaca5024f8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:20.985651 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:20.984991    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-credential-socket" pod "b0758d58-495a-4f8e-a1f6-daaca5024f8c" (UID: "b0758d58-495a-4f8e-a1f6-daaca5024f8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:20.985651 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:20.985411    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:20.985651 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:20.985432    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:20.985651 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:20.985447    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:20.985651 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:20.985460    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:20.986081 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:20.986050    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-workload-socket" pod "b0758d58-495a-4f8e-a1f6-daaca5024f8c" (UID: "b0758d58-495a-4f8e-a1f6-daaca5024f8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:20.988528 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:20.988435    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-token" pod "b0758d58-495a-4f8e-a1f6-daaca5024f8c" (UID: "b0758d58-495a-4f8e-a1f6-daaca5024f8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:20.988713 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:20.988674    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-podinfo" pod "b0758d58-495a-4f8e-a1f6-daaca5024f8c" (UID: "b0758d58-495a-4f8e-a1f6-daaca5024f8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:47:20.989804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:20.989774    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-envoy" pod "b0758d58-495a-4f8e-a1f6-daaca5024f8c" (UID: "b0758d58-495a-4f8e-a1f6-daaca5024f8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:20.991940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:20.991914    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0758d58-495a-4f8e-a1f6-daaca5024f8c-kube-api-access-72r26" pod "b0758d58-495a-4f8e-a1f6-daaca5024f8c" (UID: "b0758d58-495a-4f8e-a1f6-daaca5024f8c"). InnerVolumeSpecName "kube-api-access-72r26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:21.087520 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:21.087415    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:21.087520 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:21.087456    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:21.087520 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:21.087466    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72r26\" (UniqueName: \"kubernetes.io/projected/b0758d58-495a-4f8e-a1f6-daaca5024f8c-kube-api-access-72r26\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:21.087520 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:21.087477    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:21.087520 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:21.087486    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0758d58-495a-4f8e-a1f6-daaca5024f8c-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:21.393844 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:21.393749    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:21.787362 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:21.787315    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd" event={"ID":"b0758d58-495a-4f8e-a1f6-daaca5024f8c","Type":"ContainerDied","Data":"8472ba818549691e0db686c068747027de469ebf40a64edbc39fcde42a22d6d2"}
Oct 30 22:47:21.787890 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:21.787444    4889 scope.go:122] "RemoveContainer" containerID="ffc6f8ccb1ed86992599d55bbbe3f5a0caa42bcd4fda7cd67357c624a2bb9209"
Oct 30 22:47:21.787890 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:21.787660    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"
Oct 30 22:47:21.848421 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:21.848379    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"]
Oct 30 22:47:21.861900 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:21.861854    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-999470-openshift-default-6887745cf9-xt2fd"]
Oct 30 22:47:22.779090 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:22.779045    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0758d58-495a-4f8e-a1f6-daaca5024f8c" path="/var/lib/kubelet/pods/b0758d58-495a-4f8e-a1f6-daaca5024f8c/volumes"
Oct 30 22:47:23.171817 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:23.171721    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:25.088838 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:25.088793    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:26.755391 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:26.755326    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:26.919571 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:26.919529    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:27.111807 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:27.111712    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:28.812377 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:28.812339    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:30.763420 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:30.763378    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:30.856984 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:30.856951    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99"]
Oct 30 22:47:30.889407 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:30.889371    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp"]
Oct 30 22:47:31.053457 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:31.053369    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:31.226096 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:31.226060    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:32.563856 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:32.563819    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:34.239885 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:34.239842    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:35.820491 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:35.820455    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk"]
Oct 30 22:47:35.821380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:35.821362    4889 cpu_manager.go:414] "containerMap: removing container" podUID="b0758d58-495a-4f8e-a1f6-daaca5024f8c" containerName="istio-proxy"
Oct 30 22:47:35.821380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:35.821380    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0758d58-495a-4f8e-a1f6-daaca5024f8c" containerName="istio-proxy"
Oct 30 22:47:35.821470 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:35.821406    4889 cpu_manager.go:414] "containerMap: removing container" podUID="b0758d58-495a-4f8e-a1f6-daaca5024f8c" containerName="istio-proxy"
Oct 30 22:47:35.821470 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:35.821411    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0758d58-495a-4f8e-a1f6-daaca5024f8c" containerName="istio-proxy"
Oct 30 22:47:35.821470 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:35.821429    4889 cpu_manager.go:414] "containerMap: removing container" podUID="3b0d1da0-5e38-4000-826a-01b8110db1a1" containerName="main"
Oct 30 22:47:35.821470 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:35.821434    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b0d1da0-5e38-4000-826a-01b8110db1a1" containerName="main"
Oct 30 22:47:35.821605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:35.821561    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0758d58-495a-4f8e-a1f6-daaca5024f8c" containerName="istio-proxy"
Oct 30 22:47:35.821605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:35.821573    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0758d58-495a-4f8e-a1f6-daaca5024f8c" containerName="istio-proxy"
Oct 30 22:47:35.821605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:35.821580    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b0d1da0-5e38-4000-826a-01b8110db1a1" containerName="main"
Oct 30 22:47:35.827078 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:35.827055    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk"
Oct 30 22:47:35.844398 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:35.844359    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk"]
Oct 30 22:47:35.851419 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:35.851356    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-b03b33-serving-cert\""
Oct 30 22:47:35.858345 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:35.858300    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99" podUID="321419c3-5cd7-4e75-8646-b0573c312780" containerName="payload-processing" containerID="cri-o://42d29fbbf650ae812a835c2e28567aea00ba36ed9f3b2d9c4f8165f2780925d7" gracePeriod=25
Oct 30 22:47:35.877496 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:35.875925    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmtf2\" (UniqueName: \"kubernetes.io/projected/0997827b-80e4-4077-a22f-3d2026492158-kube-api-access-xmtf2\") pod \"maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk\" (UID: \"0997827b-80e4-4077-a22f-3d2026492158\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk"
Oct 30 22:47:35.877496 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:35.876010    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0997827b-80e4-4077-a22f-3d2026492158-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk\" (UID: \"0997827b-80e4-4077-a22f-3d2026492158\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk"
Oct 30 22:47:35.877496 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:35.876082    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0997827b-80e4-4077-a22f-3d2026492158-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk\" (UID: \"0997827b-80e4-4077-a22f-3d2026492158\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk"
Oct 30 22:47:35.877496 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:35.876276    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0997827b-80e4-4077-a22f-3d2026492158-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk\" (UID: \"0997827b-80e4-4077-a22f-3d2026492158\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk"
Oct 30 22:47:35.890099 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:35.890062    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp" podUID="ae61647a-51bb-4a9b-b51e-df9559cc9ab2" containerName="payload-pre-processing" containerID="cri-o://99951b8251a663377daeed11ca306c83691b013e1126504fdbb197365e4021d4" gracePeriod=25
Oct 30 22:47:35.961198 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:35.961143    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:35.978117 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:35.978082    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0997827b-80e4-4077-a22f-3d2026492158-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk\" (UID: \"0997827b-80e4-4077-a22f-3d2026492158\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk"
Oct 30 22:47:35.978387 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:35.978364    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmtf2\" (UniqueName: \"kubernetes.io/projected/0997827b-80e4-4077-a22f-3d2026492158-kube-api-access-xmtf2\") pod \"maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk\" (UID: \"0997827b-80e4-4077-a22f-3d2026492158\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk"
Oct 30 22:47:35.978474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:35.978421    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0997827b-80e4-4077-a22f-3d2026492158-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk\" (UID: \"0997827b-80e4-4077-a22f-3d2026492158\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk"
Oct 30 22:47:35.978537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:35.978471    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0997827b-80e4-4077-a22f-3d2026492158-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk\" (UID: \"0997827b-80e4-4077-a22f-3d2026492158\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk"
Oct 30 22:47:35.978731 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:47:35.978704    4889 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-serving-cert: secret "maas-api-e2e-shared-b-w4-b03b33-serving-cert" not found
Oct 30 22:47:35.978815 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:47:35.978802    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0997827b-80e4-4077-a22f-3d2026492158-maas-api-tls podName:0997827b-80e4-4077-a22f-3d2026492158 nodeName:}" failed. No retries permitted until 2026-10-30 22:47:36.478779841 +0000 UTC m=+2441.835146135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0997827b-80e4-4077-a22f-3d2026492158-maas-api-tls") pod "maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk" (UID: "0997827b-80e4-4077-a22f-3d2026492158") : secret "maas-api-e2e-shared-b-w4-b03b33-serving-cert" not found
Oct 30 22:47:35.978970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:35.978848    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0997827b-80e4-4077-a22f-3d2026492158-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk\" (UID: \"0997827b-80e4-4077-a22f-3d2026492158\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk"
Oct 30 22:47:35.981765 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:35.981741    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0997827b-80e4-4077-a22f-3d2026492158-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk\" (UID: \"0997827b-80e4-4077-a22f-3d2026492158\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk"
Oct 30 22:47:36.001133 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:36.001105    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmtf2\" (UniqueName: \"kubernetes.io/projected/0997827b-80e4-4077-a22f-3d2026492158-kube-api-access-xmtf2\") pod \"maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk\" (UID: \"0997827b-80e4-4077-a22f-3d2026492158\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk"
Oct 30 22:47:36.127998 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:36.127936    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:36.210464 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:36.210430    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99"
Oct 30 22:47:36.283681 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:36.283645    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/321419c3-5cd7-4e75-8646-b0573c312780-kube-api-access-6p486\" (UniqueName: \"kubernetes.io/projected/321419c3-5cd7-4e75-8646-b0573c312780-kube-api-access-6p486\") pod \"321419c3-5cd7-4e75-8646-b0573c312780\" (UID: \"321419c3-5cd7-4e75-8646-b0573c312780\") "
Oct 30 22:47:36.283893 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:36.283771    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/321419c3-5cd7-4e75-8646-b0573c312780-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321419c3-5cd7-4e75-8646-b0573c312780-plugins-config-volume\") pod \"321419c3-5cd7-4e75-8646-b0573c312780\" (UID: \"321419c3-5cd7-4e75-8646-b0573c312780\") "
Oct 30 22:47:36.284404 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:36.284367    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/321419c3-5cd7-4e75-8646-b0573c312780-plugins-config-volume" pod "321419c3-5cd7-4e75-8646-b0573c312780" (UID: "321419c3-5cd7-4e75-8646-b0573c312780"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:36.284618 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:36.284595    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321419c3-5cd7-4e75-8646-b0573c312780-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:36.286095 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:36.286070    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321419c3-5cd7-4e75-8646-b0573c312780-kube-api-access-6p486" pod "321419c3-5cd7-4e75-8646-b0573c312780" (UID: "321419c3-5cd7-4e75-8646-b0573c312780"). InnerVolumeSpecName "kube-api-access-6p486". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:36.292810 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:36.292771    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:36.385958 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:36.385871    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p486\" (UniqueName: \"kubernetes.io/projected/321419c3-5cd7-4e75-8646-b0573c312780-kube-api-access-6p486\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:36.487384 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:36.487346    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0997827b-80e4-4077-a22f-3d2026492158-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk\" (UID: \"0997827b-80e4-4077-a22f-3d2026492158\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk"
Oct 30 22:47:36.490105 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:36.490080    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0997827b-80e4-4077-a22f-3d2026492158-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk\" (UID: \"0997827b-80e4-4077-a22f-3d2026492158\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk"
Oct 30 22:47:36.745415 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:36.745381    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk"
Oct 30 22:47:36.925621 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:36.925584    4889 generic.go:356] "Generic (PLEG): container finished" podID="321419c3-5cd7-4e75-8646-b0573c312780" containerID="42d29fbbf650ae812a835c2e28567aea00ba36ed9f3b2d9c4f8165f2780925d7" exitCode=0
Oct 30 22:47:36.926130 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:36.925651    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99"
Oct 30 22:47:36.926130 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:36.925686    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99" event={"ID":"321419c3-5cd7-4e75-8646-b0573c312780","Type":"ContainerDied","Data":"42d29fbbf650ae812a835c2e28567aea00ba36ed9f3b2d9c4f8165f2780925d7"}
Oct 30 22:47:36.926130 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:36.925745    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99" event={"ID":"321419c3-5cd7-4e75-8646-b0573c312780","Type":"ContainerDied","Data":"130b5ddadfa6aff9f362248b4d251e2866835ef6b88f452e13cf969c8b2b1ca6"}
Oct 30 22:47:36.926130 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:36.925770    4889 scope.go:122] "RemoveContainer" containerID="42d29fbbf650ae812a835c2e28567aea00ba36ed9f3b2d9c4f8165f2780925d7"
Oct 30 22:47:36.938485 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:36.938460    4889 scope.go:122] "RemoveContainer" containerID="42d29fbbf650ae812a835c2e28567aea00ba36ed9f3b2d9c4f8165f2780925d7"
Oct 30 22:47:36.938874 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:47:36.938846    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42d29fbbf650ae812a835c2e28567aea00ba36ed9f3b2d9c4f8165f2780925d7\": container with ID starting with 42d29fbbf650ae812a835c2e28567aea00ba36ed9f3b2d9c4f8165f2780925d7 not found: ID does not exist" containerID="42d29fbbf650ae812a835c2e28567aea00ba36ed9f3b2d9c4f8165f2780925d7"
Oct 30 22:47:36.939011 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:36.938887    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42d29fbbf650ae812a835c2e28567aea00ba36ed9f3b2d9c4f8165f2780925d7"} err="failed to get container status \"42d29fbbf650ae812a835c2e28567aea00ba36ed9f3b2d9c4f8165f2780925d7\": rpc error: code = NotFound desc = could not find container \"42d29fbbf650ae812a835c2e28567aea00ba36ed9f3b2d9c4f8165f2780925d7\": container with ID starting with 42d29fbbf650ae812a835c2e28567aea00ba36ed9f3b2d9c4f8165f2780925d7 not found: ID does not exist"
Oct 30 22:47:36.984761 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:36.984718    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99"]
Oct 30 22:47:36.997824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:36.997726    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d85da4-7d5f448d7c-cwh99"]
Oct 30 22:47:37.110365 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:37.107347    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk"]
Oct 30 22:47:37.941001 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:37.940861    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:37.959297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:37.959193    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk" event={"ID":"0997827b-80e4-4077-a22f-3d2026492158","Type":"ContainerStarted","Data":"ce5d16e10772ddd7277ad2ad696b54f4f282bd3e724731b834d5ac486bbab865"}
Oct 30 22:47:37.959655 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:37.959311    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk" event={"ID":"0997827b-80e4-4077-a22f-3d2026492158","Type":"ContainerStarted","Data":"d1fd45c884bc97ac128b61ab45bf119ec4af3a9392ef26b145286b77796d5549"}
Oct 30 22:47:37.959655 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:37.959603    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk"
Oct 30 22:47:38.002741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:38.002674    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk" podStartSLOduration=2.627761615 podStartE2EDuration="3.002656392s" totalImagesPullingTime="374.894777ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:47:38.001903289 +0000 UTC m=+2443.358269608" watchObservedRunningTime="2026-10-30 22:47:38.002656392 +0000 UTC m=+2443.359022711"
Oct 30 22:47:38.436791 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:38.436752    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-40229e-86586c78fc-288bj"]
Oct 30 22:47:38.629059 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:38.629013    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:38.778156 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:38.778122    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="321419c3-5cd7-4e75-8646-b0573c312780" path="/var/lib/kubelet/pods/321419c3-5cd7-4e75-8646-b0573c312780/volumes"
Oct 30 22:47:38.798316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:38.798267    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:39.326961 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:39.326922    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8"]
Oct 30 22:47:39.327557 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:39.327142    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8" podUID="1c660853-ad95-4bc2-b1f5-52690a8e728f" containerName="maas-api" containerID="cri-o://4cf97bfabb8b3c3728f3644d53494e78645c1318eb3878a797f6435ecbbd7613" gracePeriod=30
Oct 30 22:47:39.444468 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:39.444434    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:39.897276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:39.895175    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8"
Oct 30 22:47:39.985041 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:39.984945    4889 generic.go:356] "Generic (PLEG): container finished" podID="1c660853-ad95-4bc2-b1f5-52690a8e728f" containerID="4cf97bfabb8b3c3728f3644d53494e78645c1318eb3878a797f6435ecbbd7613" exitCode=0
Oct 30 22:47:39.985041 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:39.985001    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8"
Oct 30 22:47:39.985041 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:39.985028    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8" event={"ID":"1c660853-ad95-4bc2-b1f5-52690a8e728f","Type":"ContainerDied","Data":"4cf97bfabb8b3c3728f3644d53494e78645c1318eb3878a797f6435ecbbd7613"}
Oct 30 22:47:39.985398 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:39.985087    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8" event={"ID":"1c660853-ad95-4bc2-b1f5-52690a8e728f","Type":"ContainerDied","Data":"548aab1d9df7dd91f13a8be98ebb88ae4c2fa658b8d73a2dc3165affe44db229"}
Oct 30 22:47:39.985398 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:39.985115    4889 scope.go:122] "RemoveContainer" containerID="4cf97bfabb8b3c3728f3644d53494e78645c1318eb3878a797f6435ecbbd7613"
Oct 30 22:47:39.998439 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:39.998418    4889 scope.go:122] "RemoveContainer" containerID="4cf97bfabb8b3c3728f3644d53494e78645c1318eb3878a797f6435ecbbd7613"
Oct 30 22:47:39.998842 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:47:39.998819    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf97bfabb8b3c3728f3644d53494e78645c1318eb3878a797f6435ecbbd7613\": container with ID starting with 4cf97bfabb8b3c3728f3644d53494e78645c1318eb3878a797f6435ecbbd7613 not found: ID does not exist" containerID="4cf97bfabb8b3c3728f3644d53494e78645c1318eb3878a797f6435ecbbd7613"
Oct 30 22:47:39.998928 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:39.998855    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf97bfabb8b3c3728f3644d53494e78645c1318eb3878a797f6435ecbbd7613"} err="failed to get container status \"4cf97bfabb8b3c3728f3644d53494e78645c1318eb3878a797f6435ecbbd7613\": rpc error: code = NotFound desc = could not find container \"4cf97bfabb8b3c3728f3644d53494e78645c1318eb3878a797f6435ecbbd7613\": container with ID starting with 4cf97bfabb8b3c3728f3644d53494e78645c1318eb3878a797f6435ecbbd7613 not found: ID does not exist"
Oct 30 22:47:40.033219 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:40.033190    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c660853-ad95-4bc2-b1f5-52690a8e728f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c660853-ad95-4bc2-b1f5-52690a8e728f-metrics-certs\") pod \"1c660853-ad95-4bc2-b1f5-52690a8e728f\" (UID: \"1c660853-ad95-4bc2-b1f5-52690a8e728f\") "
Oct 30 22:47:40.033449 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:40.033411    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c660853-ad95-4bc2-b1f5-52690a8e728f-kube-api-access-gw9nx\" (UniqueName: \"kubernetes.io/projected/1c660853-ad95-4bc2-b1f5-52690a8e728f-kube-api-access-gw9nx\") pod \"1c660853-ad95-4bc2-b1f5-52690a8e728f\" (UID: \"1c660853-ad95-4bc2-b1f5-52690a8e728f\") "
Oct 30 22:47:40.033531 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:40.033517    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c660853-ad95-4bc2-b1f5-52690a8e728f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c660853-ad95-4bc2-b1f5-52690a8e728f-openshift-service-ca\") pod \"1c660853-ad95-4bc2-b1f5-52690a8e728f\" (UID: \"1c660853-ad95-4bc2-b1f5-52690a8e728f\") "
Oct 30 22:47:40.033683 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:40.033662    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c660853-ad95-4bc2-b1f5-52690a8e728f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c660853-ad95-4bc2-b1f5-52690a8e728f-maas-api-tls\") pod \"1c660853-ad95-4bc2-b1f5-52690a8e728f\" (UID: \"1c660853-ad95-4bc2-b1f5-52690a8e728f\") "
Oct 30 22:47:40.033957 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:40.033849    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c660853-ad95-4bc2-b1f5-52690a8e728f-openshift-service-ca" pod "1c660853-ad95-4bc2-b1f5-52690a8e728f" (UID: "1c660853-ad95-4bc2-b1f5-52690a8e728f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:40.034423 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:40.034365    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c660853-ad95-4bc2-b1f5-52690a8e728f-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:40.035699 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:40.035667    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c660853-ad95-4bc2-b1f5-52690a8e728f-metrics-certs" pod "1c660853-ad95-4bc2-b1f5-52690a8e728f" (UID: "1c660853-ad95-4bc2-b1f5-52690a8e728f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:40.035791 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:40.035757    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c660853-ad95-4bc2-b1f5-52690a8e728f-maas-api-tls" pod "1c660853-ad95-4bc2-b1f5-52690a8e728f" (UID: "1c660853-ad95-4bc2-b1f5-52690a8e728f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:40.035894 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:40.035881    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c660853-ad95-4bc2-b1f5-52690a8e728f-kube-api-access-gw9nx" pod "1c660853-ad95-4bc2-b1f5-52690a8e728f" (UID: "1c660853-ad95-4bc2-b1f5-52690a8e728f"). InnerVolumeSpecName "kube-api-access-gw9nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:40.086302 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:40.086264    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:40.136237 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:40.136196    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw9nx\" (UniqueName: \"kubernetes.io/projected/1c660853-ad95-4bc2-b1f5-52690a8e728f-kube-api-access-gw9nx\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:40.136444 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:40.136289    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c660853-ad95-4bc2-b1f5-52690a8e728f-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:40.136444 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:40.136306    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c660853-ad95-4bc2-b1f5-52690a8e728f-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:40.261549 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:40.259776    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:40.317102 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:47:40.317059    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:47:40.341494 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:40.341459    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8"]
Oct 30 22:47:40.348438 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:40.348398    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d85da4-7784bfc56d-xwcw8"]
Oct 30 22:47:40.430410 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:40.430372    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:40.750182 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:40.750148    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:40.777782 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:40.777745    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c660853-ad95-4bc2-b1f5-52690a8e728f" path="/var/lib/kubelet/pods/1c660853-ad95-4bc2-b1f5-52690a8e728f/volumes"
Oct 30 22:47:42.074487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:42.074447    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"]
Oct 30 22:47:42.074987 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:42.074706    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9" podUID="ac6c9299-a77c-42a8-9b81-67f84aa9f22d" containerName="istio-proxy" containerID="cri-o://2f9ba3d90be6b7aac2a49d41918ab45c2b0a4e6158c17461b425d1218f5901f1" gracePeriod=30
Oct 30 22:47:42.137275 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:42.137188    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:43.437126 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:43.437069    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-40229e-86586c78fc-288bj" podUID="3b894efe-e301-4948-a86c-b04c0450f7d0" containerName="payload-processing" containerID="cri-o://f7a540ec0c81cfb82ac51385aded187d6c600e97e9ac3d1a04f13672c4fe182a" gracePeriod=25
Oct 30 22:47:43.951031 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:43.950885    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:43.977914 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:43.977879    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk"
Oct 30 22:47:45.456167 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:45.456128    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp" podUID="ae61647a-51bb-4a9b-b51e-df9559cc9ab2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:47:45.741366 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:45.741326    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:46.333899 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:46.333861    4889 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:47:46.334104 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:46.333927    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9" podUID="ac6c9299-a77c-42a8-9b81-67f84aa9f22d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:47:47.372907 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.372875    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:47:47.406172 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.406136    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:47.460432 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.458388    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-data\") pod \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") "
Oct 30 22:47:47.460432 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.458526    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-token\" (UniqueName: \"kubernetes.io/projected/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-token\") pod \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") "
Oct 30 22:47:47.460432 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.458603    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-workload-certs\") pod \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") "
Oct 30 22:47:47.460432 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.458635    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-workload-socket\") pod \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") "
Oct 30 22:47:47.460432 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.458735    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-podinfo\") pod \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") "
Oct 30 22:47:47.460432 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.458872    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istiod-ca-cert\") pod \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") "
Oct 30 22:47:47.460432 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.459374    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-workload-certs" pod "ac6c9299-a77c-42a8-9b81-67f84aa9f22d" (UID: "ac6c9299-a77c-42a8-9b81-67f84aa9f22d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:47.460432 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.459599    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-data" pod "ac6c9299-a77c-42a8-9b81-67f84aa9f22d" (UID: "ac6c9299-a77c-42a8-9b81-67f84aa9f22d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:47.460432 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.459719    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-credential-socket\") pod \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") "
Oct 30 22:47:47.460432 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.459789    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-workload-socket" pod "ac6c9299-a77c-42a8-9b81-67f84aa9f22d" (UID: "ac6c9299-a77c-42a8-9b81-67f84aa9f22d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:47.460432 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.459842    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-envoy\") pod \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") "
Oct 30 22:47:47.460432 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.459939    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istiod-ca-cert" pod "ac6c9299-a77c-42a8-9b81-67f84aa9f22d" (UID: "ac6c9299-a77c-42a8-9b81-67f84aa9f22d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:47.460432 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.459952    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-kube-api-access-sbtll\" (UniqueName: \"kubernetes.io/projected/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-kube-api-access-sbtll\") pod \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\" (UID: \"ac6c9299-a77c-42a8-9b81-67f84aa9f22d\") "
Oct 30 22:47:47.461352 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.460917    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:47.461352 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.460942    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:47.461352 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.460956    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:47.461352 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.460968    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:47.462446 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.461672    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-credential-socket" pod "ac6c9299-a77c-42a8-9b81-67f84aa9f22d" (UID: "ac6c9299-a77c-42a8-9b81-67f84aa9f22d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:47.465787 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.465750    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-kube-api-access-sbtll" pod "ac6c9299-a77c-42a8-9b81-67f84aa9f22d" (UID: "ac6c9299-a77c-42a8-9b81-67f84aa9f22d"). InnerVolumeSpecName "kube-api-access-sbtll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:47.465947 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.465910    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-envoy" pod "ac6c9299-a77c-42a8-9b81-67f84aa9f22d" (UID: "ac6c9299-a77c-42a8-9b81-67f84aa9f22d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:47.466218 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.466193    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-podinfo" pod "ac6c9299-a77c-42a8-9b81-67f84aa9f22d" (UID: "ac6c9299-a77c-42a8-9b81-67f84aa9f22d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:47:47.474670 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.474290    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-token" pod "ac6c9299-a77c-42a8-9b81-67f84aa9f22d" (UID: "ac6c9299-a77c-42a8-9b81-67f84aa9f22d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:47.562764 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.562728    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:47.562764 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.562761    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:47.562764 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.562771    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:47.563016 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.562781    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:47.563016 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.562791    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbtll\" (UniqueName: \"kubernetes.io/projected/ac6c9299-a77c-42a8-9b81-67f84aa9f22d-kube-api-access-sbtll\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:47.596512 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.596471    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:47.812249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:47.809513    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:48.058318 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:48.058277    4889 generic.go:356] "Generic (PLEG): container finished" podID="ac6c9299-a77c-42a8-9b81-67f84aa9f22d" containerID="2f9ba3d90be6b7aac2a49d41918ab45c2b0a4e6158c17461b425d1218f5901f1" exitCode=0
Oct 30 22:47:48.058495 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:48.058362    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"
Oct 30 22:47:48.058495 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:48.058352    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9" event={"ID":"ac6c9299-a77c-42a8-9b81-67f84aa9f22d","Type":"ContainerDied","Data":"2f9ba3d90be6b7aac2a49d41918ab45c2b0a4e6158c17461b425d1218f5901f1"}
Oct 30 22:47:48.058603 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:48.058571    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9" event={"ID":"ac6c9299-a77c-42a8-9b81-67f84aa9f22d","Type":"ContainerDied","Data":"72f9f01daf954a2c22e650426f108a448111de4c08c9fe59d0fee4832dbc7322"}
Oct 30 22:47:48.058640 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:48.058607    4889 scope.go:122] "RemoveContainer" containerID="2f9ba3d90be6b7aac2a49d41918ab45c2b0a4e6158c17461b425d1218f5901f1"
Oct 30 22:47:48.073399 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:48.073374    4889 scope.go:122] "RemoveContainer" containerID="2f9ba3d90be6b7aac2a49d41918ab45c2b0a4e6158c17461b425d1218f5901f1"
Oct 30 22:47:48.073729 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:47:48.073706    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f9ba3d90be6b7aac2a49d41918ab45c2b0a4e6158c17461b425d1218f5901f1\": container with ID starting with 2f9ba3d90be6b7aac2a49d41918ab45c2b0a4e6158c17461b425d1218f5901f1 not found: ID does not exist" containerID="2f9ba3d90be6b7aac2a49d41918ab45c2b0a4e6158c17461b425d1218f5901f1"
Oct 30 22:47:48.073807 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:48.073736    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f9ba3d90be6b7aac2a49d41918ab45c2b0a4e6158c17461b425d1218f5901f1"} err="failed to get container status \"2f9ba3d90be6b7aac2a49d41918ab45c2b0a4e6158c17461b425d1218f5901f1\": rpc error: code = NotFound desc = could not find container \"2f9ba3d90be6b7aac2a49d41918ab45c2b0a4e6158c17461b425d1218f5901f1\": container with ID starting with 2f9ba3d90be6b7aac2a49d41918ab45c2b0a4e6158c17461b425d1218f5901f1 not found: ID does not exist"
Oct 30 22:47:48.117814 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:48.117776    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"]
Oct 30 22:47:48.123352 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:48.123315    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-z58t9"]
Oct 30 22:47:48.148192 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:47:48.148154    4889 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6c9299_a77c_42a8_9b81_67f84aa9f22d.slice/crio-72f9f01daf954a2c22e650426f108a448111de4c08c9fe59d0fee4832dbc7322\": RecentStats: unable to find data in memory cache]"
Oct 30 22:47:48.733769 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:48.733725    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"]
Oct 30 22:47:48.734359 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:48.734169    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns" podUID="666b6202-2fcf-4d36-8098-deb9c3fd3718" containerName="istio-proxy" containerID="cri-o://1fe48a81d92e6664261d0400f1a1cd497211eaa83ed57ea704bdc7872ece7263" gracePeriod=30
Oct 30 22:47:48.777937 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:48.777902    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac6c9299-a77c-42a8-9b81-67f84aa9f22d" path="/var/lib/kubelet/pods/ac6c9299-a77c-42a8-9b81-67f84aa9f22d/volumes"
Oct 30 22:47:49.595725 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:49.595689    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:49.866374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:49.866286    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:50.730409 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:50.730369    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"]
Oct 30 22:47:50.819402 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:50.819367    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:50.870783 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:50.870743    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-40229e-86586c78fc-288bj" podUID="3b894efe-e301-4948-a86c-b04c0450f7d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:47:51.294147 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:51.294101    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:53.010281 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:53.009911    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:54.043638 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.043601    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:47:54.110102 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.110061    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-workload-socket\") pod \"666b6202-2fcf-4d36-8098-deb9c3fd3718\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") "
Oct 30 22:47:54.110288 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.110121    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-credential-socket\") pod \"666b6202-2fcf-4d36-8098-deb9c3fd3718\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") "
Oct 30 22:47:54.110343 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.110321    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-data\") pod \"666b6202-2fcf-4d36-8098-deb9c3fd3718\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") "
Oct 30 22:47:54.110422 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.110401    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-podinfo\") pod \"666b6202-2fcf-4d36-8098-deb9c3fd3718\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") "
Oct 30 22:47:54.110488 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.110461    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/666b6202-2fcf-4d36-8098-deb9c3fd3718-kube-api-access-65498\" (UniqueName: \"kubernetes.io/projected/666b6202-2fcf-4d36-8098-deb9c3fd3718-kube-api-access-65498\") pod \"666b6202-2fcf-4d36-8098-deb9c3fd3718\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") "
Oct 30 22:47:54.110548 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.110503    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-token\" (UniqueName: \"kubernetes.io/projected/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-token\") pod \"666b6202-2fcf-4d36-8098-deb9c3fd3718\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") "
Oct 30 22:47:54.110610 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.110553    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/666b6202-2fcf-4d36-8098-deb9c3fd3718-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666b6202-2fcf-4d36-8098-deb9c3fd3718-istiod-ca-cert\") pod \"666b6202-2fcf-4d36-8098-deb9c3fd3718\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") "
Oct 30 22:47:54.110610 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.110578    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-workload-socket" pod "666b6202-2fcf-4d36-8098-deb9c3fd3718" (UID: "666b6202-2fcf-4d36-8098-deb9c3fd3718"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:54.110610 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.110589    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-envoy\") pod \"666b6202-2fcf-4d36-8098-deb9c3fd3718\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") "
Oct 30 22:47:54.110785 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.110698    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-credential-socket" pod "666b6202-2fcf-4d36-8098-deb9c3fd3718" (UID: "666b6202-2fcf-4d36-8098-deb9c3fd3718"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:54.110785 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.110751    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-workload-certs\") pod \"666b6202-2fcf-4d36-8098-deb9c3fd3718\" (UID: \"666b6202-2fcf-4d36-8098-deb9c3fd3718\") "
Oct 30 22:47:54.111035 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.111007    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/666b6202-2fcf-4d36-8098-deb9c3fd3718-istiod-ca-cert" pod "666b6202-2fcf-4d36-8098-deb9c3fd3718" (UID: "666b6202-2fcf-4d36-8098-deb9c3fd3718"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:54.111267 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.111215    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-workload-certs" pod "666b6202-2fcf-4d36-8098-deb9c3fd3718" (UID: "666b6202-2fcf-4d36-8098-deb9c3fd3718"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:54.111582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.111561    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-data" pod "666b6202-2fcf-4d36-8098-deb9c3fd3718" (UID: "666b6202-2fcf-4d36-8098-deb9c3fd3718"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:54.111681 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.111578    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:54.111681 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.111602    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:54.111681 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.111617    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:54.111681 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.111632    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666b6202-2fcf-4d36-8098-deb9c3fd3718-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:54.113451 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.113419    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-token" pod "666b6202-2fcf-4d36-8098-deb9c3fd3718" (UID: "666b6202-2fcf-4d36-8098-deb9c3fd3718"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:54.113932 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.113890    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-envoy" pod "666b6202-2fcf-4d36-8098-deb9c3fd3718" (UID: "666b6202-2fcf-4d36-8098-deb9c3fd3718"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:54.114073 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.114053    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/666b6202-2fcf-4d36-8098-deb9c3fd3718-kube-api-access-65498" pod "666b6202-2fcf-4d36-8098-deb9c3fd3718" (UID: "666b6202-2fcf-4d36-8098-deb9c3fd3718"). InnerVolumeSpecName "kube-api-access-65498". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:54.115880 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.115852    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-podinfo" pod "666b6202-2fcf-4d36-8098-deb9c3fd3718" (UID: "666b6202-2fcf-4d36-8098-deb9c3fd3718"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:47:54.116742 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.116719    4889 generic.go:356] "Generic (PLEG): container finished" podID="666b6202-2fcf-4d36-8098-deb9c3fd3718" containerID="1fe48a81d92e6664261d0400f1a1cd497211eaa83ed57ea704bdc7872ece7263" exitCode=0
Oct 30 22:47:54.116862 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.116768    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"
Oct 30 22:47:54.116862 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.116788    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns" event={"ID":"666b6202-2fcf-4d36-8098-deb9c3fd3718","Type":"ContainerDied","Data":"1fe48a81d92e6664261d0400f1a1cd497211eaa83ed57ea704bdc7872ece7263"}
Oct 30 22:47:54.116862 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.116830    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns" event={"ID":"666b6202-2fcf-4d36-8098-deb9c3fd3718","Type":"ContainerDied","Data":"fe3259e4834d022a7fccb4926223152a8c4398adf64e8191f550231bfafd7b7b"}
Oct 30 22:47:54.116862 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.116850    4889 scope.go:122] "RemoveContainer" containerID="1fe48a81d92e6664261d0400f1a1cd497211eaa83ed57ea704bdc7872ece7263"
Oct 30 22:47:54.155582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.155555    4889 scope.go:122] "RemoveContainer" containerID="1fe48a81d92e6664261d0400f1a1cd497211eaa83ed57ea704bdc7872ece7263"
Oct 30 22:47:54.155958 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:47:54.155932    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fe48a81d92e6664261d0400f1a1cd497211eaa83ed57ea704bdc7872ece7263\": container with ID starting with 1fe48a81d92e6664261d0400f1a1cd497211eaa83ed57ea704bdc7872ece7263 not found: ID does not exist" containerID="1fe48a81d92e6664261d0400f1a1cd497211eaa83ed57ea704bdc7872ece7263"
Oct 30 22:47:54.156078 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.155970    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fe48a81d92e6664261d0400f1a1cd497211eaa83ed57ea704bdc7872ece7263"} err="failed to get container status \"1fe48a81d92e6664261d0400f1a1cd497211eaa83ed57ea704bdc7872ece7263\": rpc error: code = NotFound desc = could not find container \"1fe48a81d92e6664261d0400f1a1cd497211eaa83ed57ea704bdc7872ece7263\": container with ID starting with 1fe48a81d92e6664261d0400f1a1cd497211eaa83ed57ea704bdc7872ece7263 not found: ID does not exist"
Oct 30 22:47:54.194585 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.194546    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"]
Oct 30 22:47:54.200385 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.200339    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-zqkns"]
Oct 30 22:47:54.212741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.212702    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:54.212741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.212734    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:54.212741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.212746    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65498\" (UniqueName: \"kubernetes.io/projected/666b6202-2fcf-4d36-8098-deb9c3fd3718-kube-api-access-65498\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:54.212978 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.212761    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:54.212978 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.212774    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666b6202-2fcf-4d36-8098-deb9c3fd3718-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:54.780266 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:54.780213    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="666b6202-2fcf-4d36-8098-deb9c3fd3718" path="/var/lib/kubelet/pods/666b6202-2fcf-4d36-8098-deb9c3fd3718/volumes"
Oct 30 22:47:55.045585 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:55.045486    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:55.456181 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:55.456082    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp" podUID="ae61647a-51bb-4a9b-b51e-df9559cc9ab2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:47:56.151297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:56.151143    4889 generic.go:356] "Generic (PLEG): container finished" podID="ae61647a-51bb-4a9b-b51e-df9559cc9ab2" containerID="99951b8251a663377daeed11ca306c83691b013e1126504fdbb197365e4021d4" exitCode=0
Oct 30 22:47:56.151297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:56.151212    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp" event={"ID":"ae61647a-51bb-4a9b-b51e-df9559cc9ab2","Type":"ContainerDied","Data":"99951b8251a663377daeed11ca306c83691b013e1126504fdbb197365e4021d4"}
Oct 30 22:47:56.216528 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:56.216495    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp"
Oct 30 22:47:56.253162 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:56.253130    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae61647a-51bb-4a9b-b51e-df9559cc9ab2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae61647a-51bb-4a9b-b51e-df9559cc9ab2-plugins-config-volume\") pod \"ae61647a-51bb-4a9b-b51e-df9559cc9ab2\" (UID: \"ae61647a-51bb-4a9b-b51e-df9559cc9ab2\") "
Oct 30 22:47:56.253397 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:56.253271    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae61647a-51bb-4a9b-b51e-df9559cc9ab2-kube-api-access-m8t8g\" (UniqueName: \"kubernetes.io/projected/ae61647a-51bb-4a9b-b51e-df9559cc9ab2-kube-api-access-m8t8g\") pod \"ae61647a-51bb-4a9b-b51e-df9559cc9ab2\" (UID: \"ae61647a-51bb-4a9b-b51e-df9559cc9ab2\") "
Oct 30 22:47:56.253634 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:56.253605    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae61647a-51bb-4a9b-b51e-df9559cc9ab2-plugins-config-volume" pod "ae61647a-51bb-4a9b-b51e-df9559cc9ab2" (UID: "ae61647a-51bb-4a9b-b51e-df9559cc9ab2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:56.254020 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:56.253955    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae61647a-51bb-4a9b-b51e-df9559cc9ab2-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:56.255700 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:56.255671    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae61647a-51bb-4a9b-b51e-df9559cc9ab2-kube-api-access-m8t8g" pod "ae61647a-51bb-4a9b-b51e-df9559cc9ab2" (UID: "ae61647a-51bb-4a9b-b51e-df9559cc9ab2"). InnerVolumeSpecName "kube-api-access-m8t8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:56.355964 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:56.355925    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8t8g\" (UniqueName: \"kubernetes.io/projected/ae61647a-51bb-4a9b-b51e-df9559cc9ab2-kube-api-access-m8t8g\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:47:56.672992 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:56.672958    4889 scope.go:122] "RemoveContainer" containerID="99951b8251a663377daeed11ca306c83691b013e1126504fdbb197365e4021d4"
Oct 30 22:47:56.901293 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:56.901171    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:57.159179 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:57.159129    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp" event={"ID":"ae61647a-51bb-4a9b-b51e-df9559cc9ab2","Type":"ContainerDied","Data":"5c7e32b4f25ffc98c5e00eb644623a1481aa78a0eae80e634b860039a5446846"}
Oct 30 22:47:57.159718 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:57.159288    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp"
Oct 30 22:47:57.243723 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:57.243681    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp"]
Oct 30 22:47:57.247013 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:57.246984    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d85da4-7bdd674455-fjhgp"]
Oct 30 22:47:58.639124 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:58.639079    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:58.777452 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:58.777420    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae61647a-51bb-4a9b-b51e-df9559cc9ab2" path="/var/lib/kubelet/pods/ae61647a-51bb-4a9b-b51e-df9559cc9ab2/volumes"
Oct 30 22:47:58.842138 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:58.842096    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx"]
Oct 30 22:47:58.881264 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:58.881212    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4"]
Oct 30 22:47:59.036740 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:59.036695    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:59.206841 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:47:59.206806    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:00.433686 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:00.432620    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:00.855888 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:00.855852    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-40229e-86586c78fc-288bj" podUID="3b894efe-e301-4948-a86c-b04c0450f7d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:48:02.263954 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:02.263904    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:03.660365 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:03.660135    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:03.766088 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:03.766056    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-40229e-86586c78fc-288bj"
Oct 30 22:48:03.843083 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:03.843038    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx" podUID="8789eb93-f28c-4fca-9b60-45811622383f" containerName="payload-processing" containerID="cri-o://299a7990198fdb43910c953cdf671ddfcfcf347caca113157646f3c703beaeb3" gracePeriod=25
Oct 30 22:48:03.880609 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:03.880577    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b894efe-e301-4948-a86c-b04c0450f7d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b894efe-e301-4948-a86c-b04c0450f7d0-plugins-config-volume\") pod \"3b894efe-e301-4948-a86c-b04c0450f7d0\" (UID: \"3b894efe-e301-4948-a86c-b04c0450f7d0\") "
Oct 30 22:48:03.881191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:03.881111    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b894efe-e301-4948-a86c-b04c0450f7d0-plugins-config-volume" pod "3b894efe-e301-4948-a86c-b04c0450f7d0" (UID: "3b894efe-e301-4948-a86c-b04c0450f7d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:03.881650 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:03.881631    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b894efe-e301-4948-a86c-b04c0450f7d0-kube-api-access-lbfwm\" (UniqueName: \"kubernetes.io/projected/3b894efe-e301-4948-a86c-b04c0450f7d0-kube-api-access-lbfwm\") pod \"3b894efe-e301-4948-a86c-b04c0450f7d0\" (UID: \"3b894efe-e301-4948-a86c-b04c0450f7d0\") "
Oct 30 22:48:03.882135 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:03.882102    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4" podUID="9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81" containerName="payload-pre-processing" containerID="cri-o://585683b43d52947b987bd550f25e9986912524f74978f52873fb7f83489d7908" gracePeriod=25
Oct 30 22:48:03.883889 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:03.883865    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b894efe-e301-4948-a86c-b04c0450f7d0-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:03.889279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:03.889235    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b894efe-e301-4948-a86c-b04c0450f7d0-kube-api-access-lbfwm" pod "3b894efe-e301-4948-a86c-b04c0450f7d0" (UID: "3b894efe-e301-4948-a86c-b04c0450f7d0"). InnerVolumeSpecName "kube-api-access-lbfwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:04.011043 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:04.010985    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbfwm\" (UniqueName: \"kubernetes.io/projected/3b894efe-e301-4948-a86c-b04c0450f7d0-kube-api-access-lbfwm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:04.083867 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:04.083645    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:04.224774 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:04.224736    4889 generic.go:356] "Generic (PLEG): container finished" podID="3b894efe-e301-4948-a86c-b04c0450f7d0" containerID="f7a540ec0c81cfb82ac51385aded187d6c600e97e9ac3d1a04f13672c4fe182a" exitCode=0
Oct 30 22:48:04.224961 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:04.224789    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-40229e-86586c78fc-288bj"
Oct 30 22:48:04.224961 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:04.224814    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-40229e-86586c78fc-288bj" event={"ID":"3b894efe-e301-4948-a86c-b04c0450f7d0","Type":"ContainerDied","Data":"f7a540ec0c81cfb82ac51385aded187d6c600e97e9ac3d1a04f13672c4fe182a"}
Oct 30 22:48:04.224961 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:04.224876    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-40229e-86586c78fc-288bj" event={"ID":"3b894efe-e301-4948-a86c-b04c0450f7d0","Type":"ContainerDied","Data":"c6bd77973f04a0c7c9f6158e4df60a374968a68c6126d4238752af226e13812c"}
Oct 30 22:48:04.224961 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:04.224901    4889 scope.go:122] "RemoveContainer" containerID="f7a540ec0c81cfb82ac51385aded187d6c600e97e9ac3d1a04f13672c4fe182a"
Oct 30 22:48:04.238715 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:04.238692    4889 scope.go:122] "RemoveContainer" containerID="f7a540ec0c81cfb82ac51385aded187d6c600e97e9ac3d1a04f13672c4fe182a"
Oct 30 22:48:04.239105 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:48:04.239081    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7a540ec0c81cfb82ac51385aded187d6c600e97e9ac3d1a04f13672c4fe182a\": container with ID starting with f7a540ec0c81cfb82ac51385aded187d6c600e97e9ac3d1a04f13672c4fe182a not found: ID does not exist" containerID="f7a540ec0c81cfb82ac51385aded187d6c600e97e9ac3d1a04f13672c4fe182a"
Oct 30 22:48:04.239265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:04.239116    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7a540ec0c81cfb82ac51385aded187d6c600e97e9ac3d1a04f13672c4fe182a"} err="failed to get container status \"f7a540ec0c81cfb82ac51385aded187d6c600e97e9ac3d1a04f13672c4fe182a\": rpc error: code = NotFound desc = could not find container \"f7a540ec0c81cfb82ac51385aded187d6c600e97e9ac3d1a04f13672c4fe182a\": container with ID starting with f7a540ec0c81cfb82ac51385aded187d6c600e97e9ac3d1a04f13672c4fe182a not found: ID does not exist"
Oct 30 22:48:04.257453 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:04.257415    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:04.282959 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:04.282848    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-40229e-86586c78fc-288bj"]
Oct 30 22:48:04.296468 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:04.296433    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-40229e-86586c78fc-288bj"]
Oct 30 22:48:04.478259 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:04.468010    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:04.777481 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:04.777442    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b894efe-e301-4948-a86c-b04c0450f7d0" path="/var/lib/kubelet/pods/3b894efe-e301-4948-a86c-b04c0450f7d0/volumes"
Oct 30 22:48:04.796082 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:04.796040    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:04.961937 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:04.961859    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:05.411330 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:05.411295    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:05.754467 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:05.754429    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4" podUID="e7acdf19-b537-4edf-91f8-8b95a03a1f43" containerName="main" containerID="cri-o://7cffd4b4e65fc11cb5e4f35d643ba7be36a4986601e5cda56eca2876be5adfac" gracePeriod=45
Oct 30 22:48:06.072090 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.072057    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"
Oct 30 22:48:06.143724 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.143690    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7acdf19-b537-4edf-91f8-8b95a03a1f43-kube-api-access-dzl66\" (UniqueName: \"kubernetes.io/projected/e7acdf19-b537-4edf-91f8-8b95a03a1f43-kube-api-access-dzl66\") pod \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\" (UID: \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\") "
Oct 30 22:48:06.143950 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.143808    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-model-cache\") pod \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\" (UID: \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\") "
Oct 30 22:48:06.143950 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.143927    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-dshm\") pod \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\" (UID: \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\") "
Oct 30 22:48:06.144088 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.143996    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-tmp-dir\") pod \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\" (UID: \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\") "
Oct 30 22:48:06.144088 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.144082    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-home\" (UniqueName: \"kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-home\") pod \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\" (UID: \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\") "
Oct 30 22:48:06.144209 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.144119    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-model-cache" pod "e7acdf19-b537-4edf-91f8-8b95a03a1f43" (UID: "e7acdf19-b537-4edf-91f8-8b95a03a1f43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:06.144209 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.144165    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7acdf19-b537-4edf-91f8-8b95a03a1f43-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7acdf19-b537-4edf-91f8-8b95a03a1f43-tls-certs\") pod \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\" (UID: \"e7acdf19-b537-4edf-91f8-8b95a03a1f43\") "
Oct 30 22:48:06.144395 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.144316    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-tmp-dir" pod "e7acdf19-b537-4edf-91f8-8b95a03a1f43" (UID: "e7acdf19-b537-4edf-91f8-8b95a03a1f43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:06.144457 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.144434    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-home" pod "e7acdf19-b537-4edf-91f8-8b95a03a1f43" (UID: "e7acdf19-b537-4edf-91f8-8b95a03a1f43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:06.145129 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.145098    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:06.145129 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.145131    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:06.145396 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.145144    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:06.146537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.146494    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-dshm" pod "e7acdf19-b537-4edf-91f8-8b95a03a1f43" (UID: "e7acdf19-b537-4edf-91f8-8b95a03a1f43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:06.146626 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.146519    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7acdf19-b537-4edf-91f8-8b95a03a1f43-kube-api-access-dzl66" pod "e7acdf19-b537-4edf-91f8-8b95a03a1f43" (UID: "e7acdf19-b537-4edf-91f8-8b95a03a1f43"). InnerVolumeSpecName "kube-api-access-dzl66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:06.146915 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.146893    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7acdf19-b537-4edf-91f8-8b95a03a1f43-tls-certs" pod "e7acdf19-b537-4edf-91f8-8b95a03a1f43" (UID: "e7acdf19-b537-4edf-91f8-8b95a03a1f43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:06.246666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.246631    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7acdf19-b537-4edf-91f8-8b95a03a1f43-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:06.246666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.246662    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7acdf19-b537-4edf-91f8-8b95a03a1f43-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:06.246666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.246673    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzl66\" (UniqueName: \"kubernetes.io/projected/e7acdf19-b537-4edf-91f8-8b95a03a1f43-kube-api-access-dzl66\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:06.247861 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.247830    4889 generic.go:356] "Generic (PLEG): container finished" podID="e7acdf19-b537-4edf-91f8-8b95a03a1f43" containerID="7cffd4b4e65fc11cb5e4f35d643ba7be36a4986601e5cda56eca2876be5adfac" exitCode=0
Oct 30 22:48:06.248013 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.247884    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"
Oct 30 22:48:06.248013 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.247888    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4" event={"ID":"e7acdf19-b537-4edf-91f8-8b95a03a1f43","Type":"ContainerDied","Data":"7cffd4b4e65fc11cb5e4f35d643ba7be36a4986601e5cda56eca2876be5adfac"}
Oct 30 22:48:06.248013 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.247926    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4" event={"ID":"e7acdf19-b537-4edf-91f8-8b95a03a1f43","Type":"ContainerDied","Data":"31612fe552ea87c7f0dafceb2aab4225921b2bd15de9e57e74fd4e9b531e9e9e"}
Oct 30 22:48:06.248013 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.247947    4889 scope.go:122] "RemoveContainer" containerID="7cffd4b4e65fc11cb5e4f35d643ba7be36a4986601e5cda56eca2876be5adfac"
Oct 30 22:48:06.259979 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.259959    4889 scope.go:122] "RemoveContainer" containerID="7cffd4b4e65fc11cb5e4f35d643ba7be36a4986601e5cda56eca2876be5adfac"
Oct 30 22:48:06.260316 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:48:06.260295    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cffd4b4e65fc11cb5e4f35d643ba7be36a4986601e5cda56eca2876be5adfac\": container with ID starting with 7cffd4b4e65fc11cb5e4f35d643ba7be36a4986601e5cda56eca2876be5adfac not found: ID does not exist" containerID="7cffd4b4e65fc11cb5e4f35d643ba7be36a4986601e5cda56eca2876be5adfac"
Oct 30 22:48:06.260400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.260332    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cffd4b4e65fc11cb5e4f35d643ba7be36a4986601e5cda56eca2876be5adfac"} err="failed to get container status \"7cffd4b4e65fc11cb5e4f35d643ba7be36a4986601e5cda56eca2876be5adfac\": rpc error: code = NotFound desc = could not find container \"7cffd4b4e65fc11cb5e4f35d643ba7be36a4986601e5cda56eca2876be5adfac\": container with ID starting with 7cffd4b4e65fc11cb5e4f35d643ba7be36a4986601e5cda56eca2876be5adfac not found: ID does not exist"
Oct 30 22:48:06.305625 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.305543    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"]
Oct 30 22:48:06.309530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.309503    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4"]
Oct 30 22:48:06.658969 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.658881    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:06.784205 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:06.784155    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7acdf19-b537-4edf-91f8-8b95a03a1f43" path="/var/lib/kubelet/pods/e7acdf19-b537-4edf-91f8-8b95a03a1f43/volumes"
Oct 30 22:48:07.052313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:07.052218    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-6e0edc/test-model-6e0edc-kserve-67b89f554f-tbjc4" podUID="e7acdf19-b537-4edf-91f8-8b95a03a1f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.182:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:48:08.431345 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:08.431192    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:10.159672 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:10.159638    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:10.308037 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:48:10.307991    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:48:10.638471 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:10.638440    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:10.828964 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:10.828925    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:11.633210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:11.633173    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4" podUID="9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:48:11.633210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:11.633204    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx" podUID="8789eb93-f28c-4fca-9b60-45811622383f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:48:11.703158 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:11.702137    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:11.871821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:11.871786    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:11.956599 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:11.956515    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"]
Oct 30 22:48:11.956884 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:11.956851    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v" podUID="e2029821-7bca-434d-978b-07c37e50d52b" containerName="istio-proxy" containerID="cri-o://b52e48f34bd4d7bffe5e11b4c66c107b9157471409c9566bca7675717636caf6" gracePeriod=30
Oct 30 22:48:12.051130 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:12.051094    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:12.379034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:12.378973    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:12.562779 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:12.562743    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:12.639600 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:12.637390    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"]
Oct 30 22:48:12.639600 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:12.637664    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz" podUID="2ced92c8-b2b3-4d29-86ca-b816484d7b69" containerName="istio-proxy" containerID="cri-o://5a468b998c8581d9db3e8ccb6b740e8192310c7f102f9cc854b5d8c5162a39c2" gracePeriod=30
Oct 30 22:48:12.730994 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:12.730888    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:13.530666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:13.530625    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:13.858712 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:13.858622    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:14.039786 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:14.039557    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:14.225830 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:14.225719    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:14.807674 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:14.807625    4889 patch_prober.go:29] interesting pod/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:48:14.807845 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:14.807683    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz" podUID="2ced92c8-b2b3-4d29-86ca-b816484d7b69" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:48:15.646974 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:15.646935    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:17.264319 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.264291    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:48:17.357854 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.357821    4889 generic.go:356] "Generic (PLEG): container finished" podID="e2029821-7bca-434d-978b-07c37e50d52b" containerID="b52e48f34bd4d7bffe5e11b4c66c107b9157471409c9566bca7675717636caf6" exitCode=0
Oct 30 22:48:17.358056 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.358039    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v" event={"ID":"e2029821-7bca-434d-978b-07c37e50d52b","Type":"ContainerDied","Data":"b52e48f34bd4d7bffe5e11b4c66c107b9157471409c9566bca7675717636caf6"}
Oct 30 22:48:17.358105 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.358086    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v" event={"ID":"e2029821-7bca-434d-978b-07c37e50d52b","Type":"ContainerDied","Data":"d5644bf21076a40eba05c85f8bed879dc650261fbb8919c3fdd825d8a6428350"}
Oct 30 22:48:17.358142 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.358111    4889 scope.go:122] "RemoveContainer" containerID="b52e48f34bd4d7bffe5e11b4c66c107b9157471409c9566bca7675717636caf6"
Oct 30 22:48:17.358351 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.358336    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"
Oct 30 22:48:17.373718 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.373692    4889 scope.go:122] "RemoveContainer" containerID="b52e48f34bd4d7bffe5e11b4c66c107b9157471409c9566bca7675717636caf6"
Oct 30 22:48:17.374184 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:48:17.374152    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b52e48f34bd4d7bffe5e11b4c66c107b9157471409c9566bca7675717636caf6\": container with ID starting with b52e48f34bd4d7bffe5e11b4c66c107b9157471409c9566bca7675717636caf6 not found: ID does not exist" containerID="b52e48f34bd4d7bffe5e11b4c66c107b9157471409c9566bca7675717636caf6"
Oct 30 22:48:17.374351 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.374200    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b52e48f34bd4d7bffe5e11b4c66c107b9157471409c9566bca7675717636caf6"} err="failed to get container status \"b52e48f34bd4d7bffe5e11b4c66c107b9157471409c9566bca7675717636caf6\": rpc error: code = NotFound desc = could not find container \"b52e48f34bd4d7bffe5e11b4c66c107b9157471409c9566bca7675717636caf6\": container with ID starting with b52e48f34bd4d7bffe5e11b4c66c107b9157471409c9566bca7675717636caf6 not found: ID does not exist"
Oct 30 22:48:17.379017 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.378977    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2029821-7bca-434d-978b-07c37e50d52b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2029821-7bca-434d-978b-07c37e50d52b-istio-podinfo\") pod \"e2029821-7bca-434d-978b-07c37e50d52b\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") "
Oct 30 22:48:17.379170 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.379089    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2029821-7bca-434d-978b-07c37e50d52b-kube-api-access-74vqc\" (UniqueName: \"kubernetes.io/projected/e2029821-7bca-434d-978b-07c37e50d52b-kube-api-access-74vqc\") pod \"e2029821-7bca-434d-978b-07c37e50d52b\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") "
Oct 30 22:48:17.379170 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.379145    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-istio-data\") pod \"e2029821-7bca-434d-978b-07c37e50d52b\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") "
Oct 30 22:48:17.379616 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.379288    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-istio-envoy\") pod \"e2029821-7bca-434d-978b-07c37e50d52b\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") "
Oct 30 22:48:17.379616 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.379349    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-workload-certs\") pod \"e2029821-7bca-434d-978b-07c37e50d52b\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") "
Oct 30 22:48:17.379616 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.379443    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-credential-socket\") pod \"e2029821-7bca-434d-978b-07c37e50d52b\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") "
Oct 30 22:48:17.379616 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.379495    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-workload-socket\") pod \"e2029821-7bca-434d-978b-07c37e50d52b\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") "
Oct 30 22:48:17.379616 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.379601    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2029821-7bca-434d-978b-07c37e50d52b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2029821-7bca-434d-978b-07c37e50d52b-istiod-ca-cert\") pod \"e2029821-7bca-434d-978b-07c37e50d52b\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") "
Oct 30 22:48:17.379949 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.379675    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2029821-7bca-434d-978b-07c37e50d52b-istio-token\" (UniqueName: \"kubernetes.io/projected/e2029821-7bca-434d-978b-07c37e50d52b-istio-token\") pod \"e2029821-7bca-434d-978b-07c37e50d52b\" (UID: \"e2029821-7bca-434d-978b-07c37e50d52b\") "
Oct 30 22:48:17.380114 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.380084    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-workload-certs" pod "e2029821-7bca-434d-978b-07c37e50d52b" (UID: "e2029821-7bca-434d-978b-07c37e50d52b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:17.380733 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.380704    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-workload-socket" pod "e2029821-7bca-434d-978b-07c37e50d52b" (UID: "e2029821-7bca-434d-978b-07c37e50d52b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:17.380847 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.380727    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-credential-socket" pod "e2029821-7bca-434d-978b-07c37e50d52b" (UID: "e2029821-7bca-434d-978b-07c37e50d52b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:17.380968 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.380945    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-istio-data" pod "e2029821-7bca-434d-978b-07c37e50d52b" (UID: "e2029821-7bca-434d-978b-07c37e50d52b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:17.381056 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.381037    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2029821-7bca-434d-978b-07c37e50d52b-istiod-ca-cert" pod "e2029821-7bca-434d-978b-07c37e50d52b" (UID: "e2029821-7bca-434d-978b-07c37e50d52b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:17.381438 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.380641    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:17.382921 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.382738    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2029821-7bca-434d-978b-07c37e50d52b-kube-api-access-74vqc" pod "e2029821-7bca-434d-978b-07c37e50d52b" (UID: "e2029821-7bca-434d-978b-07c37e50d52b"). InnerVolumeSpecName "kube-api-access-74vqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:17.382921 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.382841    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2029821-7bca-434d-978b-07c37e50d52b-istio-token" pod "e2029821-7bca-434d-978b-07c37e50d52b" (UID: "e2029821-7bca-434d-978b-07c37e50d52b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:17.383530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.383504    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-istio-envoy" pod "e2029821-7bca-434d-978b-07c37e50d52b" (UID: "e2029821-7bca-434d-978b-07c37e50d52b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:17.383607 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.383530    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2029821-7bca-434d-978b-07c37e50d52b-istio-podinfo" pod "e2029821-7bca-434d-978b-07c37e50d52b" (UID: "e2029821-7bca-434d-978b-07c37e50d52b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:48:17.483330 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.483210    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74vqc\" (UniqueName: \"kubernetes.io/projected/e2029821-7bca-434d-978b-07c37e50d52b-kube-api-access-74vqc\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:17.483330 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.483270    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:17.483330 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.483280    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:17.483330 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.483288    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:17.483330 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.483297    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2029821-7bca-434d-978b-07c37e50d52b-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:17.483330 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.483306    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2029821-7bca-434d-978b-07c37e50d52b-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:17.483330 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.483315    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2029821-7bca-434d-978b-07c37e50d52b-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:17.483330 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.483323    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2029821-7bca-434d-978b-07c37e50d52b-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:17.558685 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.558639    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:17.749200 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.749159    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"]
Oct 30 22:48:17.761665 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:17.761633    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-n4d9v"]
Oct 30 22:48:18.358487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.358458    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:48:18.368139 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.368110    4889 generic.go:356] "Generic (PLEG): container finished" podID="2ced92c8-b2b3-4d29-86ca-b816484d7b69" containerID="5a468b998c8581d9db3e8ccb6b740e8192310c7f102f9cc854b5d8c5162a39c2" exitCode=0
Oct 30 22:48:18.368352 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.368155    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"
Oct 30 22:48:18.368352 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.368183    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz" event={"ID":"2ced92c8-b2b3-4d29-86ca-b816484d7b69","Type":"ContainerDied","Data":"5a468b998c8581d9db3e8ccb6b740e8192310c7f102f9cc854b5d8c5162a39c2"}
Oct 30 22:48:18.368352 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.368269    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz" event={"ID":"2ced92c8-b2b3-4d29-86ca-b816484d7b69","Type":"ContainerDied","Data":"37aa12b0e0f16e96a5530cbedd476094b19b7a8e8428d88d12c3256833b348c0"}
Oct 30 22:48:18.368352 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.368307    4889 scope.go:122] "RemoveContainer" containerID="5a468b998c8581d9db3e8ccb6b740e8192310c7f102f9cc854b5d8c5162a39c2"
Oct 30 22:48:18.382880 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.382858    4889 scope.go:122] "RemoveContainer" containerID="1d5c8fa33d8ec6ac2d3999d9ce57aed5d988289a718cc4c15c5778a4cc654c5e"
Oct 30 22:48:18.396410 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.396378    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ced92c8-b2b3-4d29-86ca-b816484d7b69-kube-api-access-sl8lx\" (UniqueName: \"kubernetes.io/projected/2ced92c8-b2b3-4d29-86ca-b816484d7b69-kube-api-access-sl8lx\") pod \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") "
Oct 30 22:48:18.396599 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.396461    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-workload-socket\") pod \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") "
Oct 30 22:48:18.396599 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.396581    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-envoy\") pod \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") "
Oct 30 22:48:18.396752 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.396634    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-credential-socket\") pod \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") "
Oct 30 22:48:18.396752 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.396662    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istiod-ca-cert\") pod \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") "
Oct 30 22:48:18.396855 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.396750    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-workload-socket" pod "2ced92c8-b2b3-4d29-86ca-b816484d7b69" (UID: "2ced92c8-b2b3-4d29-86ca-b816484d7b69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:18.396855 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.396799    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-podinfo\") pod \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") "
Oct 30 22:48:18.396971 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.396912    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-workload-certs\") pod \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") "
Oct 30 22:48:18.396971 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.396945    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-credential-socket" pod "2ced92c8-b2b3-4d29-86ca-b816484d7b69" (UID: "2ced92c8-b2b3-4d29-86ca-b816484d7b69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:18.396971 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.396966    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-token\" (UniqueName: \"kubernetes.io/projected/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-token\") pod \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") "
Oct 30 22:48:18.397139 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.397017    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-data\") pod \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\" (UID: \"2ced92c8-b2b3-4d29-86ca-b816484d7b69\") "
Oct 30 22:48:18.397301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.397275    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-workload-certs" pod "2ced92c8-b2b3-4d29-86ca-b816484d7b69" (UID: "2ced92c8-b2b3-4d29-86ca-b816484d7b69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:18.397440 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.397416    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istiod-ca-cert" pod "2ced92c8-b2b3-4d29-86ca-b816484d7b69" (UID: "2ced92c8-b2b3-4d29-86ca-b816484d7b69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:18.397507 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.397461    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-data" pod "2ced92c8-b2b3-4d29-86ca-b816484d7b69" (UID: "2ced92c8-b2b3-4d29-86ca-b816484d7b69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:18.397817 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.397792    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:18.397889 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.397823    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:18.397889 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.397837    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:18.397889 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.397850    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:18.397889 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.397865    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:18.399061 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.398890    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-envoy" pod "2ced92c8-b2b3-4d29-86ca-b816484d7b69" (UID: "2ced92c8-b2b3-4d29-86ca-b816484d7b69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:18.399061 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.399009    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ced92c8-b2b3-4d29-86ca-b816484d7b69-kube-api-access-sl8lx" pod "2ced92c8-b2b3-4d29-86ca-b816484d7b69" (UID: "2ced92c8-b2b3-4d29-86ca-b816484d7b69"). InnerVolumeSpecName "kube-api-access-sl8lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:18.399504 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.399475    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-podinfo" pod "2ced92c8-b2b3-4d29-86ca-b816484d7b69" (UID: "2ced92c8-b2b3-4d29-86ca-b816484d7b69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:48:18.399697 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.399679    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-token" pod "2ced92c8-b2b3-4d29-86ca-b816484d7b69" (UID: "2ced92c8-b2b3-4d29-86ca-b816484d7b69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:18.400250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.400215    4889 scope.go:122] "RemoveContainer" containerID="5a468b998c8581d9db3e8ccb6b740e8192310c7f102f9cc854b5d8c5162a39c2"
Oct 30 22:48:18.403970 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:48:18.403946    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a468b998c8581d9db3e8ccb6b740e8192310c7f102f9cc854b5d8c5162a39c2\": container with ID starting with 5a468b998c8581d9db3e8ccb6b740e8192310c7f102f9cc854b5d8c5162a39c2 not found: ID does not exist" containerID="5a468b998c8581d9db3e8ccb6b740e8192310c7f102f9cc854b5d8c5162a39c2"
Oct 30 22:48:18.404095 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.403979    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a468b998c8581d9db3e8ccb6b740e8192310c7f102f9cc854b5d8c5162a39c2"} err="failed to get container status \"5a468b998c8581d9db3e8ccb6b740e8192310c7f102f9cc854b5d8c5162a39c2\": rpc error: code = NotFound desc = could not find container \"5a468b998c8581d9db3e8ccb6b740e8192310c7f102f9cc854b5d8c5162a39c2\": container with ID starting with 5a468b998c8581d9db3e8ccb6b740e8192310c7f102f9cc854b5d8c5162a39c2 not found: ID does not exist"
Oct 30 22:48:18.404095 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.404004    4889 scope.go:122] "RemoveContainer" containerID="1d5c8fa33d8ec6ac2d3999d9ce57aed5d988289a718cc4c15c5778a4cc654c5e"
Oct 30 22:48:18.404344 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:48:18.404319    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d5c8fa33d8ec6ac2d3999d9ce57aed5d988289a718cc4c15c5778a4cc654c5e\": container with ID starting with 1d5c8fa33d8ec6ac2d3999d9ce57aed5d988289a718cc4c15c5778a4cc654c5e not found: ID does not exist" containerID="1d5c8fa33d8ec6ac2d3999d9ce57aed5d988289a718cc4c15c5778a4cc654c5e"
Oct 30 22:48:18.404467 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.404348    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d5c8fa33d8ec6ac2d3999d9ce57aed5d988289a718cc4c15c5778a4cc654c5e"} err="failed to get container status \"1d5c8fa33d8ec6ac2d3999d9ce57aed5d988289a718cc4c15c5778a4cc654c5e\": rpc error: code = NotFound desc = could not find container \"1d5c8fa33d8ec6ac2d3999d9ce57aed5d988289a718cc4c15c5778a4cc654c5e\": container with ID starting with 1d5c8fa33d8ec6ac2d3999d9ce57aed5d988289a718cc4c15c5778a4cc654c5e not found: ID does not exist"
Oct 30 22:48:18.487359 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.487324    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:18.499364 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.499325    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl8lx\" (UniqueName: \"kubernetes.io/projected/2ced92c8-b2b3-4d29-86ca-b816484d7b69-kube-api-access-sl8lx\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:18.499364 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.499367    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:18.499574 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.499384    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:18.499574 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.499397    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ced92c8-b2b3-4d29-86ca-b816484d7b69-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:18.660526 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.660493    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:18.720814 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.720778    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"]
Oct 30 22:48:18.734749 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.734714    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-tfzdz"]
Oct 30 22:48:18.776536 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.776500    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ced92c8-b2b3-4d29-86ca-b816484d7b69" path="/var/lib/kubelet/pods/2ced92c8-b2b3-4d29-86ca-b816484d7b69/volumes"
Oct 30 22:48:18.777123 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:18.777107    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2029821-7bca-434d-978b-07c37e50d52b" path="/var/lib/kubelet/pods/e2029821-7bca-434d-978b-07c37e50d52b/volumes"
Oct 30 22:48:19.314493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:19.314434    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:19.564680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:19.564561    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:20.717488 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.717444    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:20.764787 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.764753    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv"]
Oct 30 22:48:20.765559 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765543    4889 cpu_manager.go:414] "containerMap: removing container" podUID="321419c3-5cd7-4e75-8646-b0573c312780" containerName="payload-processing"
Oct 30 22:48:20.765638 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765560    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="321419c3-5cd7-4e75-8646-b0573c312780" containerName="payload-processing"
Oct 30 22:48:20.765638 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765574    4889 cpu_manager.go:414] "containerMap: removing container" podUID="2ced92c8-b2b3-4d29-86ca-b816484d7b69" containerName="istio-proxy"
Oct 30 22:48:20.765638 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765579    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ced92c8-b2b3-4d29-86ca-b816484d7b69" containerName="istio-proxy"
Oct 30 22:48:20.765638 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765591    4889 cpu_manager.go:414] "containerMap: removing container" podUID="e2029821-7bca-434d-978b-07c37e50d52b" containerName="istio-proxy"
Oct 30 22:48:20.765638 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765596    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2029821-7bca-434d-978b-07c37e50d52b" containerName="istio-proxy"
Oct 30 22:48:20.765638 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765611    4889 cpu_manager.go:414] "containerMap: removing container" podUID="666b6202-2fcf-4d36-8098-deb9c3fd3718" containerName="istio-proxy"
Oct 30 22:48:20.765638 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765616    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="666b6202-2fcf-4d36-8098-deb9c3fd3718" containerName="istio-proxy"
Oct 30 22:48:20.765638 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765633    4889 cpu_manager.go:414] "containerMap: removing container" podUID="ac6c9299-a77c-42a8-9b81-67f84aa9f22d" containerName="istio-proxy"
Oct 30 22:48:20.765638 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765638    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac6c9299-a77c-42a8-9b81-67f84aa9f22d" containerName="istio-proxy"
Oct 30 22:48:20.765940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765648    4889 cpu_manager.go:414] "containerMap: removing container" podUID="1c660853-ad95-4bc2-b1f5-52690a8e728f" containerName="maas-api"
Oct 30 22:48:20.765940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765653    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c660853-ad95-4bc2-b1f5-52690a8e728f" containerName="maas-api"
Oct 30 22:48:20.765940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765661    4889 cpu_manager.go:414] "containerMap: removing container" podUID="2ced92c8-b2b3-4d29-86ca-b816484d7b69" containerName="istio-proxy"
Oct 30 22:48:20.765940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765666    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ced92c8-b2b3-4d29-86ca-b816484d7b69" containerName="istio-proxy"
Oct 30 22:48:20.765940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765678    4889 cpu_manager.go:414] "containerMap: removing container" podUID="e7acdf19-b537-4edf-91f8-8b95a03a1f43" containerName="main"
Oct 30 22:48:20.765940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765683    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7acdf19-b537-4edf-91f8-8b95a03a1f43" containerName="main"
Oct 30 22:48:20.765940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765699    4889 cpu_manager.go:414] "containerMap: removing container" podUID="ae61647a-51bb-4a9b-b51e-df9559cc9ab2" containerName="payload-pre-processing"
Oct 30 22:48:20.765940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765703    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae61647a-51bb-4a9b-b51e-df9559cc9ab2" containerName="payload-pre-processing"
Oct 30 22:48:20.765940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765711    4889 cpu_manager.go:414] "containerMap: removing container" podUID="3b894efe-e301-4948-a86c-b04c0450f7d0" containerName="payload-processing"
Oct 30 22:48:20.765940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765715    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b894efe-e301-4948-a86c-b04c0450f7d0" containerName="payload-processing"
Oct 30 22:48:20.765940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765820    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7acdf19-b537-4edf-91f8-8b95a03a1f43" containerName="main"
Oct 30 22:48:20.765940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765831    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae61647a-51bb-4a9b-b51e-df9559cc9ab2" containerName="payload-pre-processing"
Oct 30 22:48:20.765940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765837    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b894efe-e301-4948-a86c-b04c0450f7d0" containerName="payload-processing"
Oct 30 22:48:20.765940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765846    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="321419c3-5cd7-4e75-8646-b0573c312780" containerName="payload-processing"
Oct 30 22:48:20.765940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765853    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ced92c8-b2b3-4d29-86ca-b816484d7b69" containerName="istio-proxy"
Oct 30 22:48:20.765940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765858    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2029821-7bca-434d-978b-07c37e50d52b" containerName="istio-proxy"
Oct 30 22:48:20.765940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765864    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="666b6202-2fcf-4d36-8098-deb9c3fd3718" containerName="istio-proxy"
Oct 30 22:48:20.765940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765872    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac6c9299-a77c-42a8-9b81-67f84aa9f22d" containerName="istio-proxy"
Oct 30 22:48:20.765940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765879    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c660853-ad95-4bc2-b1f5-52690a8e728f" containerName="maas-api"
Oct 30 22:48:20.765940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.765884    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ced92c8-b2b3-4d29-86ca-b816484d7b69" containerName="istio-proxy"
Oct 30 22:48:20.769805 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.769783    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv"
Oct 30 22:48:20.790483 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.790449    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv"]
Oct 30 22:48:20.798988 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.798940    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-bed544-dockercfg-m8wbv\""
Oct 30 22:48:20.798988 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.798974    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-bed544\""
Oct 30 22:48:20.832501 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.832452    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5031ee6-029b-44a4-8c97-68a38a1d12d4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv\" (UID: \"a5031ee6-029b-44a4-8c97-68a38a1d12d4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv"
Oct 30 22:48:20.832739 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.832592    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvn9p\" (UniqueName: \"kubernetes.io/projected/a5031ee6-029b-44a4-8c97-68a38a1d12d4-kube-api-access-tvn9p\") pod \"payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv\" (UID: \"a5031ee6-029b-44a4-8c97-68a38a1d12d4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv"
Oct 30 22:48:20.833377 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.833341    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv"]
Oct 30 22:48:20.839577 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.839545    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv"
Oct 30 22:48:20.864156 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.864117    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv"]
Oct 30 22:48:20.958337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.949254    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5031ee6-029b-44a4-8c97-68a38a1d12d4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv\" (UID: \"a5031ee6-029b-44a4-8c97-68a38a1d12d4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv"
Oct 30 22:48:20.967883 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.967655    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5031ee6-029b-44a4-8c97-68a38a1d12d4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv\" (UID: \"a5031ee6-029b-44a4-8c97-68a38a1d12d4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv"
Oct 30 22:48:20.968104 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.967877    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lktxs\" (UniqueName: \"kubernetes.io/projected/d131b55f-ea6b-4f31-8053-fe81f0c0527e-kube-api-access-lktxs\") pod \"payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv\" (UID: \"d131b55f-ea6b-4f31-8053-fe81f0c0527e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv"
Oct 30 22:48:20.968255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.968235    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d131b55f-ea6b-4f31-8053-fe81f0c0527e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv\" (UID: \"d131b55f-ea6b-4f31-8053-fe81f0c0527e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv"
Oct 30 22:48:20.968391 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.968374    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvn9p\" (UniqueName: \"kubernetes.io/projected/a5031ee6-029b-44a4-8c97-68a38a1d12d4-kube-api-access-tvn9p\") pod \"payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv\" (UID: \"a5031ee6-029b-44a4-8c97-68a38a1d12d4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv"
Oct 30 22:48:20.994264 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:20.993389    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvn9p\" (UniqueName: \"kubernetes.io/projected/a5031ee6-029b-44a4-8c97-68a38a1d12d4-kube-api-access-tvn9p\") pod \"payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv\" (UID: \"a5031ee6-029b-44a4-8c97-68a38a1d12d4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv"
Oct 30 22:48:21.005723 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:21.005313    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:21.070130 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:21.070087    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d131b55f-ea6b-4f31-8053-fe81f0c0527e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv\" (UID: \"d131b55f-ea6b-4f31-8053-fe81f0c0527e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv"
Oct 30 22:48:21.070507 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:21.070486    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lktxs\" (UniqueName: \"kubernetes.io/projected/d131b55f-ea6b-4f31-8053-fe81f0c0527e-kube-api-access-lktxs\") pod \"payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv\" (UID: \"d131b55f-ea6b-4f31-8053-fe81f0c0527e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv"
Oct 30 22:48:21.070918 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:21.070896    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d131b55f-ea6b-4f31-8053-fe81f0c0527e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv\" (UID: \"d131b55f-ea6b-4f31-8053-fe81f0c0527e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv"
Oct 30 22:48:21.088470 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:21.088436    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv"
Oct 30 22:48:21.091484 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:21.091453    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lktxs\" (UniqueName: \"kubernetes.io/projected/d131b55f-ea6b-4f31-8053-fe81f0c0527e-kube-api-access-lktxs\") pod \"payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv\" (UID: \"d131b55f-ea6b-4f31-8053-fe81f0c0527e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv"
Oct 30 22:48:21.156446 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:21.156375    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv"
Oct 30 22:48:21.401649 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:48:21.401618    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5031ee6_029b_44a4_8c97_68a38a1d12d4.slice/crio-31f4ccec5e636713a4d6be83b68cfa83405fbe2ff18facd263ab63da45983a2a WatchSource:0}: Error finding container 31f4ccec5e636713a4d6be83b68cfa83405fbe2ff18facd263ab63da45983a2a: Status 404 returned error can't find the container with id 31f4ccec5e636713a4d6be83b68cfa83405fbe2ff18facd263ab63da45983a2a
Oct 30 22:48:21.402532 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:21.402513    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv"]
Oct 30 22:48:21.503501 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:21.503472    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv"]
Oct 30 22:48:21.503615 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:48:21.503539    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd131b55f_ea6b_4f31_8053_fe81f0c0527e.slice/crio-4fd8a64bd322051827a264ac39aeb4fda4630ffc838fc4c0d894cefb981db4d2 WatchSource:0}: Error finding container 4fd8a64bd322051827a264ac39aeb4fda4630ffc838fc4c0d894cefb981db4d2: Status 404 returned error can't find the container with id 4fd8a64bd322051827a264ac39aeb4fda4630ffc838fc4c0d894cefb981db4d2
Oct 30 22:48:21.634860 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:21.634813    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4" podUID="9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:48:21.635075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:21.634836    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx" podUID="8789eb93-f28c-4fca-9b60-45811622383f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:48:21.645204 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:21.645162    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:22.413999 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:22.413962    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv" event={"ID":"d131b55f-ea6b-4f31-8053-fe81f0c0527e","Type":"ContainerStarted","Data":"1d8c057bbf32c5644bf5f293ee973b4ec7c0609ef55dfd177698287d52a94534"}
Oct 30 22:48:22.414472 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:22.414017    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv" event={"ID":"d131b55f-ea6b-4f31-8053-fe81f0c0527e","Type":"ContainerStarted","Data":"4fd8a64bd322051827a264ac39aeb4fda4630ffc838fc4c0d894cefb981db4d2"}
Oct 30 22:48:22.414472 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:22.414063    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv"
Oct 30 22:48:22.415952 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:22.415926    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv" event={"ID":"a5031ee6-029b-44a4-8c97-68a38a1d12d4","Type":"ContainerStarted","Data":"d1868142ecb0b03833ecdd7ea9169196cbf8a5f116317a138bc986035cbabda9"}
Oct 30 22:48:22.416093 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:22.415969    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv" event={"ID":"a5031ee6-029b-44a4-8c97-68a38a1d12d4","Type":"ContainerStarted","Data":"31f4ccec5e636713a4d6be83b68cfa83405fbe2ff18facd263ab63da45983a2a"}
Oct 30 22:48:22.416093 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:22.416072    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv"
Oct 30 22:48:22.465926 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:22.465868    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv" podStartSLOduration=2.465851403 podStartE2EDuration="2.465851403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:22.462909963 +0000 UTC m=+2487.819276282" watchObservedRunningTime="2026-10-30 22:48:22.465851403 +0000 UTC m=+2487.822217718"
Oct 30 22:48:22.506315 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:22.506244    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv" podStartSLOduration=2.506202853 podStartE2EDuration="2.506202853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:22.503352127 +0000 UTC m=+2487.859718442" watchObservedRunningTime="2026-10-30 22:48:22.506202853 +0000 UTC m=+2487.862569169"
Oct 30 22:48:23.442512 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:23.441562    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:23.956136 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:23.956104    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5"]
Oct 30 22:48:24.164478 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.164424    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:24.273757 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.273729    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx"
Oct 30 22:48:24.277927 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.277899    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4"
Oct 30 22:48:24.353397 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.353367    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8789eb93-f28c-4fca-9b60-45811622383f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8789eb93-f28c-4fca-9b60-45811622383f-plugins-config-volume\") pod \"8789eb93-f28c-4fca-9b60-45811622383f\" (UID: \"8789eb93-f28c-4fca-9b60-45811622383f\") "
Oct 30 22:48:24.353620 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.353450    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8789eb93-f28c-4fca-9b60-45811622383f-kube-api-access-cztlc\" (UniqueName: \"kubernetes.io/projected/8789eb93-f28c-4fca-9b60-45811622383f-kube-api-access-cztlc\") pod \"8789eb93-f28c-4fca-9b60-45811622383f\" (UID: \"8789eb93-f28c-4fca-9b60-45811622383f\") "
Oct 30 22:48:24.353620 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.353608    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81-plugins-config-volume\") pod \"9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81\" (UID: \"9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81\") "
Oct 30 22:48:24.353759 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.353636    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81-kube-api-access-hzxcl\" (UniqueName: \"kubernetes.io/projected/9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81-kube-api-access-hzxcl\") pod \"9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81\" (UID: \"9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81\") "
Oct 30 22:48:24.353961 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.353846    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8789eb93-f28c-4fca-9b60-45811622383f-plugins-config-volume" pod "8789eb93-f28c-4fca-9b60-45811622383f" (UID: "8789eb93-f28c-4fca-9b60-45811622383f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:24.354091 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.354066    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81-plugins-config-volume" pod "9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81" (UID: "9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:24.354359 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.354336    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:24.354359 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.354360    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8789eb93-f28c-4fca-9b60-45811622383f-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:24.356295 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.356265    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81-kube-api-access-hzxcl" pod "9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81" (UID: "9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81"). InnerVolumeSpecName "kube-api-access-hzxcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:24.356632 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.356590    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8789eb93-f28c-4fca-9b60-45811622383f-kube-api-access-cztlc" pod "8789eb93-f28c-4fca-9b60-45811622383f" (UID: "8789eb93-f28c-4fca-9b60-45811622383f"). InnerVolumeSpecName "kube-api-access-cztlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:24.369414 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.369365    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:24.443616 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.443523    4889 generic.go:356] "Generic (PLEG): container finished" podID="8789eb93-f28c-4fca-9b60-45811622383f" containerID="299a7990198fdb43910c953cdf671ddfcfcf347caca113157646f3c703beaeb3" exitCode=0
Oct 30 22:48:24.443616 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.443572    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx"
Oct 30 22:48:24.444094 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.443608    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx" event={"ID":"8789eb93-f28c-4fca-9b60-45811622383f","Type":"ContainerDied","Data":"299a7990198fdb43910c953cdf671ddfcfcf347caca113157646f3c703beaeb3"}
Oct 30 22:48:24.444094 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.443666    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx" event={"ID":"8789eb93-f28c-4fca-9b60-45811622383f","Type":"ContainerDied","Data":"320d3156c8e39f6fc8123f5eb0d263d902ca7a51b4766b338ec608ddd19b6fee"}
Oct 30 22:48:24.444094 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.443693    4889 scope.go:122] "RemoveContainer" containerID="299a7990198fdb43910c953cdf671ddfcfcf347caca113157646f3c703beaeb3"
Oct 30 22:48:24.445511 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.445477    4889 generic.go:356] "Generic (PLEG): container finished" podID="9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81" containerID="585683b43d52947b987bd550f25e9986912524f74978f52873fb7f83489d7908" exitCode=0
Oct 30 22:48:24.445656 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.445558    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4"
Oct 30 22:48:24.445656 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.445588    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4" event={"ID":"9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81","Type":"ContainerDied","Data":"585683b43d52947b987bd550f25e9986912524f74978f52873fb7f83489d7908"}
Oct 30 22:48:24.445656 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.445622    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4" event={"ID":"9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81","Type":"ContainerDied","Data":"15bfd3af80e24e4bada8f1b816baf9b4f1d2f8bcdfb4297316d3aa034cfed5b0"}
Oct 30 22:48:24.456365 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.456178    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cztlc\" (UniqueName: \"kubernetes.io/projected/8789eb93-f28c-4fca-9b60-45811622383f-kube-api-access-cztlc\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:24.456365 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.456206    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzxcl\" (UniqueName: \"kubernetes.io/projected/9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81-kube-api-access-hzxcl\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:24.457198 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.457175    4889 scope.go:122] "RemoveContainer" containerID="299a7990198fdb43910c953cdf671ddfcfcf347caca113157646f3c703beaeb3"
Oct 30 22:48:24.457634 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:48:24.457608    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"299a7990198fdb43910c953cdf671ddfcfcf347caca113157646f3c703beaeb3\": container with ID starting with 299a7990198fdb43910c953cdf671ddfcfcf347caca113157646f3c703beaeb3 not found: ID does not exist" containerID="299a7990198fdb43910c953cdf671ddfcfcf347caca113157646f3c703beaeb3"
Oct 30 22:48:24.457713 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.457648    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"299a7990198fdb43910c953cdf671ddfcfcf347caca113157646f3c703beaeb3"} err="failed to get container status \"299a7990198fdb43910c953cdf671ddfcfcf347caca113157646f3c703beaeb3\": rpc error: code = NotFound desc = could not find container \"299a7990198fdb43910c953cdf671ddfcfcf347caca113157646f3c703beaeb3\": container with ID starting with 299a7990198fdb43910c953cdf671ddfcfcf347caca113157646f3c703beaeb3 not found: ID does not exist"
Oct 30 22:48:24.457713 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.457672    4889 scope.go:122] "RemoveContainer" containerID="585683b43d52947b987bd550f25e9986912524f74978f52873fb7f83489d7908"
Oct 30 22:48:24.473837 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.473801    4889 scope.go:122] "RemoveContainer" containerID="585683b43d52947b987bd550f25e9986912524f74978f52873fb7f83489d7908"
Oct 30 22:48:24.474263 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:48:24.474238    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"585683b43d52947b987bd550f25e9986912524f74978f52873fb7f83489d7908\": container with ID starting with 585683b43d52947b987bd550f25e9986912524f74978f52873fb7f83489d7908 not found: ID does not exist" containerID="585683b43d52947b987bd550f25e9986912524f74978f52873fb7f83489d7908"
Oct 30 22:48:24.474378 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.474285    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"585683b43d52947b987bd550f25e9986912524f74978f52873fb7f83489d7908"} err="failed to get container status \"585683b43d52947b987bd550f25e9986912524f74978f52873fb7f83489d7908\": rpc error: code = NotFound desc = could not find container \"585683b43d52947b987bd550f25e9986912524f74978f52873fb7f83489d7908\": container with ID starting with 585683b43d52947b987bd550f25e9986912524f74978f52873fb7f83489d7908 not found: ID does not exist"
Oct 30 22:48:24.500825 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.500790    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4"]
Oct 30 22:48:24.507280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.507242    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f0bb45-64bb8c9b67-2dxq4"]
Oct 30 22:48:24.543544 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.543509    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx"]
Oct 30 22:48:24.547662 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.547635    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f0bb45-5f54c4488f-hdmsx"]
Oct 30 22:48:24.776836 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.776800    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8789eb93-f28c-4fca-9b60-45811622383f" path="/var/lib/kubelet/pods/8789eb93-f28c-4fca-9b60-45811622383f/volumes"
Oct 30 22:48:24.777321 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:24.777306    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81" path="/var/lib/kubelet/pods/9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81/volumes"
Oct 30 22:48:25.135461 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:25.135356    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:25.309577 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:25.309379    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:27.093194 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:27.093161    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:28.806025 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:28.805987    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:28.956803 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:28.956756    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5" podUID="bf02b0cb-f674-4854-af66-93ecd22bc3a4" containerName="payload-processing" containerID="cri-o://8e23212e6789198052b2a0f5cfb8ddf88cb9561a96150a20ffc15f9d9e3623b9" gracePeriod=25
Oct 30 22:48:29.253512 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:29.253481    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5"
Oct 30 22:48:29.363417 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:29.363318    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf02b0cb-f674-4854-af66-93ecd22bc3a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf02b0cb-f674-4854-af66-93ecd22bc3a4-plugins-config-volume\") pod \"bf02b0cb-f674-4854-af66-93ecd22bc3a4\" (UID: \"bf02b0cb-f674-4854-af66-93ecd22bc3a4\") "
Oct 30 22:48:29.363616 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:29.363487    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf02b0cb-f674-4854-af66-93ecd22bc3a4-kube-api-access-fghmj\" (UniqueName: \"kubernetes.io/projected/bf02b0cb-f674-4854-af66-93ecd22bc3a4-kube-api-access-fghmj\") pod \"bf02b0cb-f674-4854-af66-93ecd22bc3a4\" (UID: \"bf02b0cb-f674-4854-af66-93ecd22bc3a4\") "
Oct 30 22:48:29.363816 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:29.363788    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf02b0cb-f674-4854-af66-93ecd22bc3a4-plugins-config-volume" pod "bf02b0cb-f674-4854-af66-93ecd22bc3a4" (UID: "bf02b0cb-f674-4854-af66-93ecd22bc3a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:29.364136 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:29.364118    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf02b0cb-f674-4854-af66-93ecd22bc3a4-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:29.366024 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:29.366003    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf02b0cb-f674-4854-af66-93ecd22bc3a4-kube-api-access-fghmj" pod "bf02b0cb-f674-4854-af66-93ecd22bc3a4" (UID: "bf02b0cb-f674-4854-af66-93ecd22bc3a4"). InnerVolumeSpecName "kube-api-access-fghmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:29.465460 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:29.465421    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fghmj\" (UniqueName: \"kubernetes.io/projected/bf02b0cb-f674-4854-af66-93ecd22bc3a4-kube-api-access-fghmj\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:29.494728 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:29.494694    4889 generic.go:356] "Generic (PLEG): container finished" podID="bf02b0cb-f674-4854-af66-93ecd22bc3a4" containerID="8e23212e6789198052b2a0f5cfb8ddf88cb9561a96150a20ffc15f9d9e3623b9" exitCode=0
Oct 30 22:48:29.494914 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:29.494733    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5" event={"ID":"bf02b0cb-f674-4854-af66-93ecd22bc3a4","Type":"ContainerDied","Data":"8e23212e6789198052b2a0f5cfb8ddf88cb9561a96150a20ffc15f9d9e3623b9"}
Oct 30 22:48:29.494914 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:29.494802    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5" event={"ID":"bf02b0cb-f674-4854-af66-93ecd22bc3a4","Type":"ContainerDied","Data":"3b33230477f85ecaa7997a4339c8a7899ae245c933f68bb663951813a8d09b0e"}
Oct 30 22:48:29.494914 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:29.494743    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5"
Oct 30 22:48:29.494914 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:29.494833    4889 scope.go:122] "RemoveContainer" containerID="8e23212e6789198052b2a0f5cfb8ddf88cb9561a96150a20ffc15f9d9e3623b9"
Oct 30 22:48:29.506966 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:29.506945    4889 scope.go:122] "RemoveContainer" containerID="8e23212e6789198052b2a0f5cfb8ddf88cb9561a96150a20ffc15f9d9e3623b9"
Oct 30 22:48:29.507370 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:48:29.507347    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e23212e6789198052b2a0f5cfb8ddf88cb9561a96150a20ffc15f9d9e3623b9\": container with ID starting with 8e23212e6789198052b2a0f5cfb8ddf88cb9561a96150a20ffc15f9d9e3623b9 not found: ID does not exist" containerID="8e23212e6789198052b2a0f5cfb8ddf88cb9561a96150a20ffc15f9d9e3623b9"
Oct 30 22:48:29.507434 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:29.507381    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e23212e6789198052b2a0f5cfb8ddf88cb9561a96150a20ffc15f9d9e3623b9"} err="failed to get container status \"8e23212e6789198052b2a0f5cfb8ddf88cb9561a96150a20ffc15f9d9e3623b9\": rpc error: code = NotFound desc = could not find container \"8e23212e6789198052b2a0f5cfb8ddf88cb9561a96150a20ffc15f9d9e3623b9\": container with ID starting with 8e23212e6789198052b2a0f5cfb8ddf88cb9561a96150a20ffc15f9d9e3623b9 not found: ID does not exist"
Oct 30 22:48:29.552289 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:29.552246    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5"]
Oct 30 22:48:29.558437 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:29.558401    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c13db0-5dbfdd689-q5zt5"]
Oct 30 22:48:30.288833 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:30.288796    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:30.454305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:30.454264    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:30.625469 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:30.625381    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:30.816600 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:30.816568    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf02b0cb-f674-4854-af66-93ecd22bc3a4" path="/var/lib/kubelet/pods/bf02b0cb-f674-4854-af66-93ecd22bc3a4/volumes"
Oct 30 22:48:32.109481 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:32.109445    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:32.685279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:32.685239    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:32.868794 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:32.868759    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:33.313117 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:33.313082    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:33.436704 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:33.436673    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv"
Oct 30 22:48:33.436872 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:33.436780    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv"
Oct 30 22:48:34.718887 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:34.718844    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:35.301892 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:35.301853    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"]
Oct 30 22:48:35.410764 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:35.410726    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:36.686341 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:36.686308    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:38.548412 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:38.548341    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:39.708002 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:39.707959    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:39.874641 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:39.874604    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:39.982178 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:39.982068    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"]
Oct 30 22:48:39.982769 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:39.982707    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4" podUID="19ea5d7a-f7db-40f2-ad56-66e429ea7393" containerName="istio-proxy" containerID="cri-o://9261a037fe2a36e8d8bcb499ad03cf04c47319124f7e1600acd02ef43b9044e9" gracePeriod=30
Oct 30 22:48:40.098971 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:40.098932    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:40.304260 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:48:40.304204    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:48:40.337396 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:40.337348    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:42.133386 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:42.133350    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:43.806216 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:43.806174    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:45.286349 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.286319    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:48:45.350767 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.350725    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-envoy\") pod \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") "
Oct 30 22:48:45.350976 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.350787    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-podinfo\") pod \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") "
Oct 30 22:48:45.350976 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.350845    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istiod-ca-cert\") pod \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") "
Oct 30 22:48:45.350976 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.350896    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-credential-socket\") pod \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") "
Oct 30 22:48:45.350976 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.350926    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ea5d7a-f7db-40f2-ad56-66e429ea7393-kube-api-access-z8rmv\" (UniqueName: \"kubernetes.io/projected/19ea5d7a-f7db-40f2-ad56-66e429ea7393-kube-api-access-z8rmv\") pod \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") "
Oct 30 22:48:45.350976 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.350967    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-workload-certs\") pod \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") "
Oct 30 22:48:45.351294 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.351147    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-data\") pod \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") "
Oct 30 22:48:45.351366 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.351349    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-token\" (UniqueName: \"kubernetes.io/projected/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-token\") pod \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") "
Oct 30 22:48:45.352283 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.351416    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-workload-socket\") pod \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\" (UID: \"19ea5d7a-f7db-40f2-ad56-66e429ea7393\") "
Oct 30 22:48:45.352283 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.352041    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-credential-socket" pod "19ea5d7a-f7db-40f2-ad56-66e429ea7393" (UID: "19ea5d7a-f7db-40f2-ad56-66e429ea7393"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:45.352474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.352398    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istiod-ca-cert" pod "19ea5d7a-f7db-40f2-ad56-66e429ea7393" (UID: "19ea5d7a-f7db-40f2-ad56-66e429ea7393"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:45.352600 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.352577    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-data" pod "19ea5d7a-f7db-40f2-ad56-66e429ea7393" (UID: "19ea5d7a-f7db-40f2-ad56-66e429ea7393"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:45.352876 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.352811    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-workload-certs" pod "19ea5d7a-f7db-40f2-ad56-66e429ea7393" (UID: "19ea5d7a-f7db-40f2-ad56-66e429ea7393"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:45.353023 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.353002    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-workload-socket" pod "19ea5d7a-f7db-40f2-ad56-66e429ea7393" (UID: "19ea5d7a-f7db-40f2-ad56-66e429ea7393"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:45.355517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.355486    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-token" pod "19ea5d7a-f7db-40f2-ad56-66e429ea7393" (UID: "19ea5d7a-f7db-40f2-ad56-66e429ea7393"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:45.355665 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.355538    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ea5d7a-f7db-40f2-ad56-66e429ea7393-kube-api-access-z8rmv" pod "19ea5d7a-f7db-40f2-ad56-66e429ea7393" (UID: "19ea5d7a-f7db-40f2-ad56-66e429ea7393"). InnerVolumeSpecName "kube-api-access-z8rmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:45.355665 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.355610    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-podinfo" pod "19ea5d7a-f7db-40f2-ad56-66e429ea7393" (UID: "19ea5d7a-f7db-40f2-ad56-66e429ea7393"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:48:45.356016 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.355990    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-envoy" pod "19ea5d7a-f7db-40f2-ad56-66e429ea7393" (UID: "19ea5d7a-f7db-40f2-ad56-66e429ea7393"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:45.453469 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.453426    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:45.453469 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.453460    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:45.453469 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.453471    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:45.453469 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.453480    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:45.453820 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.453490    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8rmv\" (UniqueName: \"kubernetes.io/projected/19ea5d7a-f7db-40f2-ad56-66e429ea7393-kube-api-access-z8rmv\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:45.453820 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.453501    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:45.453820 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.453509    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:45.453820 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.453516    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ea5d7a-f7db-40f2-ad56-66e429ea7393-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:45.453820 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.453524    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ea5d7a-f7db-40f2-ad56-66e429ea7393-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:45.507519 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.507205    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:45.637647 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.637614    4889 generic.go:356] "Generic (PLEG): container finished" podID="19ea5d7a-f7db-40f2-ad56-66e429ea7393" containerID="9261a037fe2a36e8d8bcb499ad03cf04c47319124f7e1600acd02ef43b9044e9" exitCode=0
Oct 30 22:48:45.637881 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.637671    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"
Oct 30 22:48:45.637881 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.637700    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4" event={"ID":"19ea5d7a-f7db-40f2-ad56-66e429ea7393","Type":"ContainerDied","Data":"9261a037fe2a36e8d8bcb499ad03cf04c47319124f7e1600acd02ef43b9044e9"}
Oct 30 22:48:45.637881 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.637771    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4" event={"ID":"19ea5d7a-f7db-40f2-ad56-66e429ea7393","Type":"ContainerDied","Data":"a3bbacb841d9848828b0226d5c477215e4832f732969f3c7774a47ed47fbbc9b"}
Oct 30 22:48:45.637881 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.637801    4889 scope.go:122] "RemoveContainer" containerID="9261a037fe2a36e8d8bcb499ad03cf04c47319124f7e1600acd02ef43b9044e9"
Oct 30 22:48:45.651321 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.651297    4889 scope.go:122] "RemoveContainer" containerID="9261a037fe2a36e8d8bcb499ad03cf04c47319124f7e1600acd02ef43b9044e9"
Oct 30 22:48:45.651668 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:48:45.651647    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9261a037fe2a36e8d8bcb499ad03cf04c47319124f7e1600acd02ef43b9044e9\": container with ID starting with 9261a037fe2a36e8d8bcb499ad03cf04c47319124f7e1600acd02ef43b9044e9 not found: ID does not exist" containerID="9261a037fe2a36e8d8bcb499ad03cf04c47319124f7e1600acd02ef43b9044e9"
Oct 30 22:48:45.651724 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.651675    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9261a037fe2a36e8d8bcb499ad03cf04c47319124f7e1600acd02ef43b9044e9"} err="failed to get container status \"9261a037fe2a36e8d8bcb499ad03cf04c47319124f7e1600acd02ef43b9044e9\": rpc error: code = NotFound desc = could not find container \"9261a037fe2a36e8d8bcb499ad03cf04c47319124f7e1600acd02ef43b9044e9\": container with ID starting with 9261a037fe2a36e8d8bcb499ad03cf04c47319124f7e1600acd02ef43b9044e9 not found: ID does not exist"
Oct 30 22:48:45.697188 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.697152    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"]
Oct 30 22:48:45.709921 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:45.709887    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-w4hj4"]
Oct 30 22:48:46.777106 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:46.777069    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19ea5d7a-f7db-40f2-ad56-66e429ea7393" path="/var/lib/kubelet/pods/19ea5d7a-f7db-40f2-ad56-66e429ea7393/volumes"
Oct 30 22:48:47.042600 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.042505    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"]
Oct 30 22:48:47.043348 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.043329    4889 cpu_manager.go:414] "containerMap: removing container" podUID="9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81" containerName="payload-pre-processing"
Oct 30 22:48:47.043402 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.043349    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81" containerName="payload-pre-processing"
Oct 30 22:48:47.043402 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.043364    4889 cpu_manager.go:414] "containerMap: removing container" podUID="19ea5d7a-f7db-40f2-ad56-66e429ea7393" containerName="istio-proxy"
Oct 30 22:48:47.043402 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.043370    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19ea5d7a-f7db-40f2-ad56-66e429ea7393" containerName="istio-proxy"
Oct 30 22:48:47.043402 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.043402    4889 cpu_manager.go:414] "containerMap: removing container" podUID="8789eb93-f28c-4fca-9b60-45811622383f" containerName="payload-processing"
Oct 30 22:48:47.043541 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.043407    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8789eb93-f28c-4fca-9b60-45811622383f" containerName="payload-processing"
Oct 30 22:48:47.043541 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.043423    4889 cpu_manager.go:414] "containerMap: removing container" podUID="bf02b0cb-f674-4854-af66-93ecd22bc3a4" containerName="payload-processing"
Oct 30 22:48:47.043541 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.043428    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf02b0cb-f674-4854-af66-93ecd22bc3a4" containerName="payload-processing"
Oct 30 22:48:47.043639 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.043561    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af1fe41-3e2a-4beb-8f28-0e6a3ecbdb81" containerName="payload-pre-processing"
Oct 30 22:48:47.043639 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.043571    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="19ea5d7a-f7db-40f2-ad56-66e429ea7393" containerName="istio-proxy"
Oct 30 22:48:47.043639 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.043582    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="8789eb93-f28c-4fca-9b60-45811622383f" containerName="payload-processing"
Oct 30 22:48:47.043639 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.043589    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf02b0cb-f674-4854-af66-93ecd22bc3a4" containerName="payload-processing"
Oct 30 22:48:47.049285 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.049258    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"
Oct 30 22:48:47.067188 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.067150    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"]
Oct 30 22:48:47.075497 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.075467    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-c511a3\"/\"openshift-service-ca.crt\""
Oct 30 22:48:47.075673 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.075510    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-c511a3\"/\"kube-root-ca.crt\""
Oct 30 22:48:47.075673 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.075602    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-c511a3\"/\"default-dockercfg-nnnnt\""
Oct 30 22:48:47.081285 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.081258    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-c511a3\"/\"auth-test-model-c511a3-kserve-self-signed-certs\""
Oct 30 22:48:47.189543 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.189511    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-tmp-dir\") pod \"auth-test-model-c511a3-kserve-6656df857c-hlbzj\" (UID: \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\") " pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"
Oct 30 22:48:47.189753 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.189714    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-dshm\") pod \"auth-test-model-c511a3-kserve-6656df857c-hlbzj\" (UID: \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\") " pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"
Oct 30 22:48:47.189832 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.189766    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-model-cache\") pod \"auth-test-model-c511a3-kserve-6656df857c-hlbzj\" (UID: \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\") " pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"
Oct 30 22:48:47.189888 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.189833    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe7f6368-5fa6-4f71-8dab-9283198e9c13-tls-certs\") pod \"auth-test-model-c511a3-kserve-6656df857c-hlbzj\" (UID: \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\") " pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"
Oct 30 22:48:47.189957 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.189937    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg4ct\" (UniqueName: \"kubernetes.io/projected/fe7f6368-5fa6-4f71-8dab-9283198e9c13-kube-api-access-gg4ct\") pod \"auth-test-model-c511a3-kserve-6656df857c-hlbzj\" (UID: \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\") " pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"
Oct 30 22:48:47.190088 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.190064    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-home\") pod \"auth-test-model-c511a3-kserve-6656df857c-hlbzj\" (UID: \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\") " pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"
Oct 30 22:48:47.198789 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.198755    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:47.291581 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.291541    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg4ct\" (UniqueName: \"kubernetes.io/projected/fe7f6368-5fa6-4f71-8dab-9283198e9c13-kube-api-access-gg4ct\") pod \"auth-test-model-c511a3-kserve-6656df857c-hlbzj\" (UID: \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\") " pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"
Oct 30 22:48:47.291778 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.291629    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-home\") pod \"auth-test-model-c511a3-kserve-6656df857c-hlbzj\" (UID: \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\") " pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"
Oct 30 22:48:47.291778 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.291659    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-tmp-dir\") pod \"auth-test-model-c511a3-kserve-6656df857c-hlbzj\" (UID: \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\") " pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"
Oct 30 22:48:47.291778 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.291766    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-dshm\") pod \"auth-test-model-c511a3-kserve-6656df857c-hlbzj\" (UID: \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\") " pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"
Oct 30 22:48:47.291987 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.291813    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-model-cache\") pod \"auth-test-model-c511a3-kserve-6656df857c-hlbzj\" (UID: \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\") " pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"
Oct 30 22:48:47.291987 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.291831    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe7f6368-5fa6-4f71-8dab-9283198e9c13-tls-certs\") pod \"auth-test-model-c511a3-kserve-6656df857c-hlbzj\" (UID: \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\") " pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"
Oct 30 22:48:47.292128 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.292104    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-home\") pod \"auth-test-model-c511a3-kserve-6656df857c-hlbzj\" (UID: \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\") " pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"
Oct 30 22:48:47.292169 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.292128    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-tmp-dir\") pod \"auth-test-model-c511a3-kserve-6656df857c-hlbzj\" (UID: \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\") " pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"
Oct 30 22:48:47.292261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.292205    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-model-cache\") pod \"auth-test-model-c511a3-kserve-6656df857c-hlbzj\" (UID: \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\") " pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"
Oct 30 22:48:47.294417 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.294357    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-dshm\") pod \"auth-test-model-c511a3-kserve-6656df857c-hlbzj\" (UID: \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\") " pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"
Oct 30 22:48:47.294715 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.294689    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe7f6368-5fa6-4f71-8dab-9283198e9c13-tls-certs\") pod \"auth-test-model-c511a3-kserve-6656df857c-hlbzj\" (UID: \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\") " pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"
Oct 30 22:48:47.319104 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.319069    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg4ct\" (UniqueName: \"kubernetes.io/projected/fe7f6368-5fa6-4f71-8dab-9283198e9c13-kube-api-access-gg4ct\") pod \"auth-test-model-c511a3-kserve-6656df857c-hlbzj\" (UID: \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\") " pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"
Oct 30 22:48:47.367248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.366284    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"
Oct 30 22:48:47.378638 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.377960    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:47.704103 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:47.704007    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"]
Oct 30 22:48:47.709163 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:48:47.709130    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe7f6368_5fa6_4f71_8dab_9283198e9c13.slice/crio-386fd50543c06d275d3fa256e02ec1d04fad0c309b3cf2861831e20cad549892 WatchSource:0}: Error finding container 386fd50543c06d275d3fa256e02ec1d04fad0c309b3cf2861831e20cad549892: Status 404 returned error can't find the container with id 386fd50543c06d275d3fa256e02ec1d04fad0c309b3cf2861831e20cad549892
Oct 30 22:48:48.680132 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:48.680090    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj" event={"ID":"fe7f6368-5fa6-4f71-8dab-9283198e9c13","Type":"ContainerStarted","Data":"8aac8fc244bee4779d34f2cc88d7f9b01148959ca17c7c3a4a912617845d5995"}
Oct 30 22:48:48.680586 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:48.680148    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj" event={"ID":"fe7f6368-5fa6-4f71-8dab-9283198e9c13","Type":"ContainerStarted","Data":"386fd50543c06d275d3fa256e02ec1d04fad0c309b3cf2861831e20cad549892"}
Oct 30 22:48:48.725462 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:48.725400    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj" podStartSLOduration=1.7253865419999999 podStartE2EDuration="1.725386542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:48.722582259 +0000 UTC m=+2514.078948575" watchObservedRunningTime="2026-10-30 22:48:48.725386542 +0000 UTC m=+2514.081752857"
Oct 30 22:48:49.144823 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:49.144781    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:49.367355 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:49.367320    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"
Oct 30 22:48:49.829896 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:49.829841    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:50.336310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.336000    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49" podUID="31fba7f5-e583-4c1f-af77-01d86ec53b57" containerName="main" containerID="cri-o://5bd41712edcea971bb34b98f8e5cb1ae528e3f75fda5d7ca0dc9ef990a4f771f" gracePeriod=45
Oct 30 22:48:50.448970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.448930    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49" podUID="31fba7f5-e583-4c1f-af77-01d86ec53b57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.181:8000/ready\": dial tcp 10.132.0.181:8000: connect: connection refused"
Oct 30 22:48:50.677537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.677505    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"
Oct 30 22:48:50.699301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.699272    4889 generic.go:356] "Generic (PLEG): container finished" podID="31fba7f5-e583-4c1f-af77-01d86ec53b57" containerID="5bd41712edcea971bb34b98f8e5cb1ae528e3f75fda5d7ca0dc9ef990a4f771f" exitCode=0
Oct 30 22:48:50.699524 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.699329    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"
Oct 30 22:48:50.699524 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.699360    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49" event={"ID":"31fba7f5-e583-4c1f-af77-01d86ec53b57","Type":"ContainerDied","Data":"5bd41712edcea971bb34b98f8e5cb1ae528e3f75fda5d7ca0dc9ef990a4f771f"}
Oct 30 22:48:50.699524 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.699404    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49" event={"ID":"31fba7f5-e583-4c1f-af77-01d86ec53b57","Type":"ContainerDied","Data":"a59a7e55a87989473e5688eeab2eb2a69292ac206cfac1dcfb2ac03f37e3d7fc"}
Oct 30 22:48:50.699524 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.699421    4889 scope.go:122] "RemoveContainer" containerID="5bd41712edcea971bb34b98f8e5cb1ae528e3f75fda5d7ca0dc9ef990a4f771f"
Oct 30 22:48:50.736467 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.735502    4889 scope.go:122] "RemoveContainer" containerID="5bd41712edcea971bb34b98f8e5cb1ae528e3f75fda5d7ca0dc9ef990a4f771f"
Oct 30 22:48:50.736902 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:48:50.736825    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bd41712edcea971bb34b98f8e5cb1ae528e3f75fda5d7ca0dc9ef990a4f771f\": container with ID starting with 5bd41712edcea971bb34b98f8e5cb1ae528e3f75fda5d7ca0dc9ef990a4f771f not found: ID does not exist" containerID="5bd41712edcea971bb34b98f8e5cb1ae528e3f75fda5d7ca0dc9ef990a4f771f"
Oct 30 22:48:50.736902 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.736869    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bd41712edcea971bb34b98f8e5cb1ae528e3f75fda5d7ca0dc9ef990a4f771f"} err="failed to get container status \"5bd41712edcea971bb34b98f8e5cb1ae528e3f75fda5d7ca0dc9ef990a4f771f\": rpc error: code = NotFound desc = could not find container \"5bd41712edcea971bb34b98f8e5cb1ae528e3f75fda5d7ca0dc9ef990a4f771f\": container with ID starting with 5bd41712edcea971bb34b98f8e5cb1ae528e3f75fda5d7ca0dc9ef990a4f771f not found: ID does not exist"
Oct 30 22:48:50.756843 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.756810    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31fba7f5-e583-4c1f-af77-01d86ec53b57-kube-api-access-sdpgf\" (UniqueName: \"kubernetes.io/projected/31fba7f5-e583-4c1f-af77-01d86ec53b57-kube-api-access-sdpgf\") pod \"31fba7f5-e583-4c1f-af77-01d86ec53b57\" (UID: \"31fba7f5-e583-4c1f-af77-01d86ec53b57\") "
Oct 30 22:48:50.757057 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.756870    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-model-cache\") pod \"31fba7f5-e583-4c1f-af77-01d86ec53b57\" (UID: \"31fba7f5-e583-4c1f-af77-01d86ec53b57\") "
Oct 30 22:48:50.757057 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.757035    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-home\" (UniqueName: \"kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-home\") pod \"31fba7f5-e583-4c1f-af77-01d86ec53b57\" (UID: \"31fba7f5-e583-4c1f-af77-01d86ec53b57\") "
Oct 30 22:48:50.757194 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.757132    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31fba7f5-e583-4c1f-af77-01d86ec53b57-tls-certs\" (UniqueName: \"kubernetes.io/secret/31fba7f5-e583-4c1f-af77-01d86ec53b57-tls-certs\") pod \"31fba7f5-e583-4c1f-af77-01d86ec53b57\" (UID: \"31fba7f5-e583-4c1f-af77-01d86ec53b57\") "
Oct 30 22:48:50.757293 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.757200    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-model-cache" pod "31fba7f5-e583-4c1f-af77-01d86ec53b57" (UID: "31fba7f5-e583-4c1f-af77-01d86ec53b57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:50.757410 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.757388    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-home" pod "31fba7f5-e583-4c1f-af77-01d86ec53b57" (UID: "31fba7f5-e583-4c1f-af77-01d86ec53b57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:50.757569 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.757545    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-dshm\" (UniqueName: \"kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-dshm\") pod \"31fba7f5-e583-4c1f-af77-01d86ec53b57\" (UID: \"31fba7f5-e583-4c1f-af77-01d86ec53b57\") "
Oct 30 22:48:50.757569 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.757622    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-tmp-dir\") pod \"31fba7f5-e583-4c1f-af77-01d86ec53b57\" (UID: \"31fba7f5-e583-4c1f-af77-01d86ec53b57\") "
Oct 30 22:48:50.757925 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.757894    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-tmp-dir" pod "31fba7f5-e583-4c1f-af77-01d86ec53b57" (UID: "31fba7f5-e583-4c1f-af77-01d86ec53b57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:50.758932 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.758897    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:50.759044 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.758940    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:50.759044 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.758958    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:50.759573 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.759542    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31fba7f5-e583-4c1f-af77-01d86ec53b57-tls-certs" pod "31fba7f5-e583-4c1f-af77-01d86ec53b57" (UID: "31fba7f5-e583-4c1f-af77-01d86ec53b57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:50.759661 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.759605    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31fba7f5-e583-4c1f-af77-01d86ec53b57-kube-api-access-sdpgf" pod "31fba7f5-e583-4c1f-af77-01d86ec53b57" (UID: "31fba7f5-e583-4c1f-af77-01d86ec53b57"). InnerVolumeSpecName "kube-api-access-sdpgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:50.759889 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.759870    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-dshm" pod "31fba7f5-e583-4c1f-af77-01d86ec53b57" (UID: "31fba7f5-e583-4c1f-af77-01d86ec53b57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:50.860973 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.860815    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31fba7f5-e583-4c1f-af77-01d86ec53b57-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:50.860973 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.860846    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31fba7f5-e583-4c1f-af77-01d86ec53b57-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:50.860973 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.860856    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdpgf\" (UniqueName: \"kubernetes.io/projected/31fba7f5-e583-4c1f-af77-01d86ec53b57-kube-api-access-sdpgf\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:48:50.978727 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:50.978627    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:51.052197 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:51.052141    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"]
Oct 30 22:48:51.056412 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:51.056375    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-bc48cc/test-model-bc48cc-kserve-599cd96549-wfl49"]
Oct 30 22:48:52.777494 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:52.777457    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31fba7f5-e583-4c1f-af77-01d86ec53b57" path="/var/lib/kubelet/pods/31fba7f5-e583-4c1f-af77-01d86ec53b57/volumes"
Oct 30 22:48:57.367481 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:57.367436    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"
Oct 30 22:48:57.383186 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:57.383154    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"
Oct 30 22:48:57.840347 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:57.840310    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:57.911563 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:57.911533    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"
Oct 30 22:48:58.017275 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:58.017210    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:58.733257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:58.733201    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs"]
Oct 30 22:48:58.734065 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:58.734049    4889 cpu_manager.go:414] "containerMap: removing container" podUID="31fba7f5-e583-4c1f-af77-01d86ec53b57" containerName="main"
Oct 30 22:48:58.734111 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:58.734066    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31fba7f5-e583-4c1f-af77-01d86ec53b57" containerName="main"
Oct 30 22:48:58.734196 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:58.734186    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="31fba7f5-e583-4c1f-af77-01d86ec53b57" containerName="main"
Oct 30 22:48:58.739779 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:58.739753    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs"
Oct 30 22:48:58.767478 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:58.759974    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs"]
Oct 30 22:48:58.771718 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:58.771676    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-bed544-serving-cert\""
Oct 30 22:48:58.863211 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:58.863158    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac794af8-6cd6-473f-bee9-3ea457729889-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs\" (UID: \"ac794af8-6cd6-473f-bee9-3ea457729889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs"
Oct 30 22:48:58.863394 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:58.863299    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac794af8-6cd6-473f-bee9-3ea457729889-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs\" (UID: \"ac794af8-6cd6-473f-bee9-3ea457729889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs"
Oct 30 22:48:58.863394 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:58.863356    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdnwd\" (UniqueName: \"kubernetes.io/projected/ac794af8-6cd6-473f-bee9-3ea457729889-kube-api-access-hdnwd\") pod \"maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs\" (UID: \"ac794af8-6cd6-473f-bee9-3ea457729889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs"
Oct 30 22:48:58.863545 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:58.863525    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac794af8-6cd6-473f-bee9-3ea457729889-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs\" (UID: \"ac794af8-6cd6-473f-bee9-3ea457729889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs"
Oct 30 22:48:58.882338 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:58.882301    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:58.965707 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:58.965660    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac794af8-6cd6-473f-bee9-3ea457729889-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs\" (UID: \"ac794af8-6cd6-473f-bee9-3ea457729889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs"
Oct 30 22:48:58.966043 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:58.965774    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac794af8-6cd6-473f-bee9-3ea457729889-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs\" (UID: \"ac794af8-6cd6-473f-bee9-3ea457729889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs"
Oct 30 22:48:58.966043 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:58.965860    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdnwd\" (UniqueName: \"kubernetes.io/projected/ac794af8-6cd6-473f-bee9-3ea457729889-kube-api-access-hdnwd\") pod \"maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs\" (UID: \"ac794af8-6cd6-473f-bee9-3ea457729889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs"
Oct 30 22:48:58.966043 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:58.966012    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac794af8-6cd6-473f-bee9-3ea457729889-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs\" (UID: \"ac794af8-6cd6-473f-bee9-3ea457729889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs"
Oct 30 22:48:58.967443 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:58.967352    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac794af8-6cd6-473f-bee9-3ea457729889-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs\" (UID: \"ac794af8-6cd6-473f-bee9-3ea457729889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs"
Oct 30 22:48:58.970281 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:58.970205    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac794af8-6cd6-473f-bee9-3ea457729889-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs\" (UID: \"ac794af8-6cd6-473f-bee9-3ea457729889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs"
Oct 30 22:48:58.970557 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:58.970251    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac794af8-6cd6-473f-bee9-3ea457729889-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs\" (UID: \"ac794af8-6cd6-473f-bee9-3ea457729889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs"
Oct 30 22:48:58.988793 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:58.988684    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdnwd\" (UniqueName: \"kubernetes.io/projected/ac794af8-6cd6-473f-bee9-3ea457729889-kube-api-access-hdnwd\") pod \"maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs\" (UID: \"ac794af8-6cd6-473f-bee9-3ea457729889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs"
Oct 30 22:48:59.050808 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:59.050754    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:59.058681 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:59.058642    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs"
Oct 30 22:48:59.214698 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:59.214647    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:59.372906 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:59.372862    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs"]
Oct 30 22:48:59.373076 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:48:59.373039    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac794af8_6cd6_473f_bee9_3ea457729889.slice/crio-19fef0741fa185e1f7c045a164eae2db656c3b0d128be9543f1f380adb7e56e2 WatchSource:0}: Error finding container 19fef0741fa185e1f7c045a164eae2db656c3b0d128be9543f1f380adb7e56e2: Status 404 returned error can't find the container with id 19fef0741fa185e1f7c045a164eae2db656c3b0d128be9543f1f380adb7e56e2
Oct 30 22:48:59.922195 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:59.922093    4889 generic.go:356] "Generic (PLEG): container finished" podID="efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9" containerID="472db4676bd2b485e4f867a6a49d2aad2e7babee5603cd02514bcfefb1a8bee6" exitCode=0
Oct 30 22:48:59.922195 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:59.922170    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd" event={"ID":"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9","Type":"ContainerDied","Data":"472db4676bd2b485e4f867a6a49d2aad2e7babee5603cd02514bcfefb1a8bee6"}
Oct 30 22:48:59.922829 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:59.922807    4889 scope.go:122] "RemoveContainer" containerID="472db4676bd2b485e4f867a6a49d2aad2e7babee5603cd02514bcfefb1a8bee6"
Oct 30 22:48:59.924421 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:59.924392    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs" event={"ID":"ac794af8-6cd6-473f-bee9-3ea457729889","Type":"ContainerStarted","Data":"6204d8fc1cde0c397b2b2602aa4c0cdab2e2317c849be2d534b09904e3a56a2b"}
Oct 30 22:48:59.924523 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:59.924447    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs" event={"ID":"ac794af8-6cd6-473f-bee9-3ea457729889","Type":"ContainerStarted","Data":"19fef0741fa185e1f7c045a164eae2db656c3b0d128be9543f1f380adb7e56e2"}
Oct 30 22:48:59.924523 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:59.924475    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs"
Oct 30 22:48:59.925148 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:59.925117    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:48:59.925236 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:59.925196    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:48:59.925296 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:48:59.925282    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:49:00.028421 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:00.027012    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs" podStartSLOduration=1.7503830169999999 podStartE2EDuration="2.026992156s" totalImagesPullingTime="276.609139ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:00.02488984 +0000 UTC m=+2525.381256157" watchObservedRunningTime="2026-10-30 22:49:00.026992156 +0000 UTC m=+2525.383358473"
Oct 30 22:49:00.390573 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:00.390538    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:00.944027 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:00.943992    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd" event={"ID":"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9","Type":"ContainerStarted","Data":"b1fbe6a13a71825e5dee3039efea5f663115f26bfd42233ee2845ac6bdf56ed1"}
Oct 30 22:49:00.950414 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:00.950385    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:00.972841 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:00.972805    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:49:01.128272 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:01.127624    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:01.298104 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:01.298071    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:01.371134 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:01.371097    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:49:01.574002 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:01.573910    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:01.973629 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:01.973532    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 22:49:01.973629 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:01.973598    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd" podUID="efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 22:49:02.159161 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:02.159111    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:02.334252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:02.331837    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:02.543250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:02.542843    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:02.972976 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:02.972938    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 22:49:02.973186 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:02.972997    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd" podUID="efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 22:49:03.102408 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:03.102359    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:03.972148 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:03.972115    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 22:49:03.972369 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:03.972174    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd" podUID="efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 22:49:04.972809 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:04.972769    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 22:49:04.973367 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:04.972827    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd" podUID="efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 22:49:05.096987 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:05.096955    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:05.790804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:05.790765    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"]
Oct 30 22:49:05.795205 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:05.795179    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"
Oct 30 22:49:05.840254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:05.828191    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-6f2eba\"/\"default-dockercfg-cfdgd\""
Oct 30 22:49:05.840254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:05.828959    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-6f2eba\"/\"auth-test-model-6f2eba-kserve-self-signed-certs\""
Oct 30 22:49:05.842056 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:05.841862    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-6f2eba\"/\"openshift-service-ca.crt\""
Oct 30 22:49:05.842462 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:05.841869    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-6f2eba\"/\"kube-root-ca.crt\""
Oct 30 22:49:05.850306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:05.850277    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"]
Oct 30 22:49:05.904086 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:05.904053    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:05.932253 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:05.909908    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-dshm\") pod \"auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj\" (UID: \"fa03e689-d506-45ca-9883-6627c13d25da\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"
Oct 30 22:49:05.932783 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:05.932748    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-tmp-dir\") pod \"auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj\" (UID: \"fa03e689-d506-45ca-9883-6627c13d25da\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"
Oct 30 22:49:05.934685 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:05.934630    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-model-cache\") pod \"auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj\" (UID: \"fa03e689-d506-45ca-9883-6627c13d25da\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"
Oct 30 22:49:05.934852 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:05.934700    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjkgq\" (UniqueName: \"kubernetes.io/projected/fa03e689-d506-45ca-9883-6627c13d25da-kube-api-access-zjkgq\") pod \"auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj\" (UID: \"fa03e689-d506-45ca-9883-6627c13d25da\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"
Oct 30 22:49:05.934852 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:05.934765    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa03e689-d506-45ca-9883-6627c13d25da-tls-certs\") pod \"auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj\" (UID: \"fa03e689-d506-45ca-9883-6627c13d25da\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"
Oct 30 22:49:05.934852 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:05.934825    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-home\") pod \"auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj\" (UID: \"fa03e689-d506-45ca-9883-6627c13d25da\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"
Oct 30 22:49:05.950194 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:05.950165    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs"
Oct 30 22:49:05.973070 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:05.973029    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 22:49:05.973583 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:05.973095    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd" podUID="efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 22:49:06.066307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:06.062466    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-home\") pod \"auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj\" (UID: \"fa03e689-d506-45ca-9883-6627c13d25da\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"
Oct 30 22:49:06.066307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:06.062558    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-dshm\") pod \"auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj\" (UID: \"fa03e689-d506-45ca-9883-6627c13d25da\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"
Oct 30 22:49:06.066307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:06.062686    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-tmp-dir\") pod \"auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj\" (UID: \"fa03e689-d506-45ca-9883-6627c13d25da\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"
Oct 30 22:49:06.066307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:06.062963    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-model-cache\") pod \"auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj\" (UID: \"fa03e689-d506-45ca-9883-6627c13d25da\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"
Oct 30 22:49:06.066307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:06.062998    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjkgq\" (UniqueName: \"kubernetes.io/projected/fa03e689-d506-45ca-9883-6627c13d25da-kube-api-access-zjkgq\") pod \"auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj\" (UID: \"fa03e689-d506-45ca-9883-6627c13d25da\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"
Oct 30 22:49:06.066307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:06.063047    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa03e689-d506-45ca-9883-6627c13d25da-tls-certs\") pod \"auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj\" (UID: \"fa03e689-d506-45ca-9883-6627c13d25da\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"
Oct 30 22:49:06.071605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:06.071570    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-home\") pod \"auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj\" (UID: \"fa03e689-d506-45ca-9883-6627c13d25da\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"
Oct 30 22:49:06.084561 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:06.084523    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-model-cache\") pod \"auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj\" (UID: \"fa03e689-d506-45ca-9883-6627c13d25da\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"
Oct 30 22:49:06.090258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:06.089287    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-tmp-dir\") pod \"auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj\" (UID: \"fa03e689-d506-45ca-9883-6627c13d25da\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"
Oct 30 22:49:06.099460 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:06.098877    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjkgq\" (UniqueName: \"kubernetes.io/projected/fa03e689-d506-45ca-9883-6627c13d25da-kube-api-access-zjkgq\") pod \"auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj\" (UID: \"fa03e689-d506-45ca-9883-6627c13d25da\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"
Oct 30 22:49:06.100876 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:06.100808    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa03e689-d506-45ca-9883-6627c13d25da-tls-certs\") pod \"auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj\" (UID: \"fa03e689-d506-45ca-9883-6627c13d25da\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"
Oct 30 22:49:06.103982 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:06.103861    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-dshm\") pod \"auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj\" (UID: \"fa03e689-d506-45ca-9883-6627c13d25da\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"
Oct 30 22:49:06.148317 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:06.148274    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"
Oct 30 22:49:06.499405 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:49:06.499374    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa03e689_d506_45ca_9883_6627c13d25da.slice/crio-5e0ec6ba8453767dbd11b313264b224cadda7eed2ddd4b9d08c0efcb34021bbb WatchSource:0}: Error finding container 5e0ec6ba8453767dbd11b313264b224cadda7eed2ddd4b9d08c0efcb34021bbb: Status 404 returned error can't find the container with id 5e0ec6ba8453767dbd11b313264b224cadda7eed2ddd4b9d08c0efcb34021bbb
Oct 30 22:49:06.500424 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:06.500378    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"]
Oct 30 22:49:06.904990 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:06.904892    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:06.977876 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:06.977845    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:49:06.979306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:06.979286    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:49:06.996297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:06.996259    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj" event={"ID":"fa03e689-d506-45ca-9883-6627c13d25da","Type":"ContainerStarted","Data":"085638291a60cfb021f15c2f0354c6941029dc83a3e8eb7b0564854167490d95"}
Oct 30 22:49:06.996515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:06.996307    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj" event={"ID":"fa03e689-d506-45ca-9883-6627c13d25da","Type":"ContainerStarted","Data":"5e0ec6ba8453767dbd11b313264b224cadda7eed2ddd4b9d08c0efcb34021bbb"}
Oct 30 22:49:07.121203 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:07.118659    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj" podStartSLOduration=2.118629967 podStartE2EDuration="2.118629967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:07.111600375 +0000 UTC m=+2532.467966689" watchObservedRunningTime="2026-10-30 22:49:07.118629967 +0000 UTC m=+2532.474996282"
Oct 30 22:49:07.148887 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:07.148844    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"
Oct 30 22:49:08.182296 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:08.182253    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:08.353195 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:08.353163    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:08.411846 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:08.411812    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"]
Oct 30 22:49:08.412128 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:08.412077    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln" podUID="ffbcb47e-8cd4-4746-b9b2-ff257d8cde01" containerName="istio-proxy" containerID="cri-o://37dc7fac3bb494743a71969dbfdb8f9f3fe7782c663a4c20bf85f912070d4f9e" gracePeriod=30
Oct 30 22:49:08.547830 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:08.547792    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:08.718202 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:08.718160    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:09.189943 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.189904    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"]
Oct 30 22:49:09.193744 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.193722    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.213043 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.213011    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"]
Oct 30 22:49:09.225323 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.221752    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-bc48cc-openshift-default-dockercfg-h45f9\""
Oct 30 22:49:09.236583 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.236539    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-podinfo\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.236815 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.236715    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d85wn\" (UniqueName: \"kubernetes.io/projected/b877606e-a40b-4013-97a3-5087c7b47ce6-kube-api-access-d85wn\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.236815 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.236763    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-envoy\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.236815 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.236801    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-workload-certs\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.237029 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.237011    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b877606e-a40b-4013-97a3-5087c7b47ce6-istiod-ca-cert\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.237097 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.237053    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-data\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.237149 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.237126    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-workload-socket\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.237209 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.237189    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-token\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.237397 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.237372    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-credential-socket\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.338386 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.338338    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d85wn\" (UniqueName: \"kubernetes.io/projected/b877606e-a40b-4013-97a3-5087c7b47ce6-kube-api-access-d85wn\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.338386 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.338389    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-envoy\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.338657 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.338411    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-workload-certs\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.338657 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.338508    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b877606e-a40b-4013-97a3-5087c7b47ce6-istiod-ca-cert\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.338657 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.338542    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-data\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.338657 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.338588    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-workload-socket\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.338657 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.338636    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-token\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.338934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.338662    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-credential-socket\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.338934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.338758    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-podinfo\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.339051 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.338972    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-data\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.339111 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.339090    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-workload-socket\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.339175 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.339156    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-workload-certs\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.339376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.339262    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-credential-socket\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.339460 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.339407    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b877606e-a40b-4013-97a3-5087c7b47ce6-istiod-ca-cert\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.341032 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.341008    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-envoy\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.341350 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.341331    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-podinfo\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.359856 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.359815    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-token\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.360042 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.359986    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d85wn\" (UniqueName: \"kubernetes.io/projected/b877606e-a40b-4013-97a3-5087c7b47ce6-kube-api-access-d85wn\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.437465 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.437421    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:09.514727 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.514687    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:09.625143 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.616587    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:09.866085 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:49:09.866050    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb877606e_a40b_4013_97a3_5087c7b47ce6.slice/crio-d55f464344ba7b249bcac4bf1f0a66294d5595d95e23eb02b153bad6699dc15b WatchSource:0}: Error finding container d55f464344ba7b249bcac4bf1f0a66294d5595d95e23eb02b153bad6699dc15b: Status 404 returned error can't find the container with id d55f464344ba7b249bcac4bf1f0a66294d5595d95e23eb02b153bad6699dc15b
Oct 30 22:49:09.866673 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.866649    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"]
Oct 30 22:49:09.868958 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.868928    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:49:09.869030 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.868996    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:49:09.869030 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:09.869022    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:49:10.024940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:10.024900    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs" event={"ID":"b877606e-a40b-4013-97a3-5087c7b47ce6","Type":"ContainerStarted","Data":"42454bc96dc4435253654aff08064138cb9daabfa54173de405b8080e9dd634d"}
Oct 30 22:49:10.025168 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:10.024961    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs" event={"ID":"b877606e-a40b-4013-97a3-5087c7b47ce6","Type":"ContainerStarted","Data":"d55f464344ba7b249bcac4bf1f0a66294d5595d95e23eb02b153bad6699dc15b"}
Oct 30 22:49:10.083260 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:10.081403    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs" podStartSLOduration=1.08137801 podStartE2EDuration="1.08137801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:10.071960765 +0000 UTC m=+2535.428327079" watchObservedRunningTime="2026-10-30 22:49:10.08137801 +0000 UTC m=+2535.437744328"
Oct 30 22:49:10.312949 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:49:10.312902    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:49:10.525255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:10.519049    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:10.525255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:10.521817    4889 patch_prober.go:29] interesting pod/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Oct 30 22:49:10.525255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:10.521898    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs" podUID="b877606e-a40b-4013-97a3-5087c7b47ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Oct 30 22:49:10.540254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:10.536099    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:11.357767 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:11.357723    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"]
Oct 30 22:49:11.358212 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:11.357937    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz" podUID="9fca6fa2-01cd-4f49-96c3-60da13e5c889" containerName="istio-proxy" containerID="cri-o://73c14c5ce12b23c158a3d4542fce534c4b13e1577f8d541bdffa24311c0883b6" gracePeriod=30
Oct 30 22:49:11.534284 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:11.532683    4889 patch_prober.go:29] interesting pod/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Oct 30 22:49:11.534284 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:11.532770    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs" podUID="b877606e-a40b-4013-97a3-5087c7b47ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Oct 30 22:49:11.992373 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:11.992311    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:12.530388 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:12.530348    4889 patch_prober.go:29] interesting pod/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Oct 30 22:49:12.531200 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:12.531162    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs" podUID="b877606e-a40b-4013-97a3-5087c7b47ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Oct 30 22:49:12.648962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:12.648404    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:13.518392 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.517807    4889 patch_prober.go:29] interesting pod/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Oct 30 22:49:13.518392 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.517874    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs" podUID="b877606e-a40b-4013-97a3-5087c7b47ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Oct 30 22:49:13.802216 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.802188    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:49:13.933258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.920881    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istiod-ca-cert\") pod \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") "
Oct 30 22:49:13.938258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.937612    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-token\" (UniqueName: \"kubernetes.io/projected/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-token\") pod \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") "
Oct 30 22:49:13.938537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.938515    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-envoy\") pod \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") "
Oct 30 22:49:13.939309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.939287    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-data\") pod \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") "
Oct 30 22:49:13.939942 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.939923    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-podinfo\") pod \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") "
Oct 30 22:49:13.940493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.940474    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-workload-socket\") pod \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") "
Oct 30 22:49:13.941532 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.941503    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-workload-certs\") pod \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") "
Oct 30 22:49:13.941640 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.941567    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-kube-api-access-qvrrc\" (UniqueName: \"kubernetes.io/projected/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-kube-api-access-qvrrc\") pod \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") "
Oct 30 22:49:13.942313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.941715    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-credential-socket\") pod \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\" (UID: \"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01\") "
Oct 30 22:49:13.942313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.937462    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istiod-ca-cert" pod "ffbcb47e-8cd4-4746-b9b2-ff257d8cde01" (UID: "ffbcb47e-8cd4-4746-b9b2-ff257d8cde01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:13.942313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.939790    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-data" pod "ffbcb47e-8cd4-4746-b9b2-ff257d8cde01" (UID: "ffbcb47e-8cd4-4746-b9b2-ff257d8cde01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:13.942313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.940860    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-workload-socket" pod "ffbcb47e-8cd4-4746-b9b2-ff257d8cde01" (UID: "ffbcb47e-8cd4-4746-b9b2-ff257d8cde01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:13.942313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.941827    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-envoy" pod "ffbcb47e-8cd4-4746-b9b2-ff257d8cde01" (UID: "ffbcb47e-8cd4-4746-b9b2-ff257d8cde01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:13.942313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.941978    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-workload-certs" pod "ffbcb47e-8cd4-4746-b9b2-ff257d8cde01" (UID: "ffbcb47e-8cd4-4746-b9b2-ff257d8cde01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:13.942313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.942271    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-credential-socket" pod "ffbcb47e-8cd4-4746-b9b2-ff257d8cde01" (UID: "ffbcb47e-8cd4-4746-b9b2-ff257d8cde01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:13.942857 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.942821    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-token" pod "ffbcb47e-8cd4-4746-b9b2-ff257d8cde01" (UID: "ffbcb47e-8cd4-4746-b9b2-ff257d8cde01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:13.944467 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.944425    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-kube-api-access-qvrrc" pod "ffbcb47e-8cd4-4746-b9b2-ff257d8cde01" (UID: "ffbcb47e-8cd4-4746-b9b2-ff257d8cde01"). InnerVolumeSpecName "kube-api-access-qvrrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:13.945417 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.945395    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-podinfo" pod "ffbcb47e-8cd4-4746-b9b2-ff257d8cde01" (UID: "ffbcb47e-8cd4-4746-b9b2-ff257d8cde01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:49:13.951651 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.951621    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:13.951857 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.951839    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvrrc\" (UniqueName: \"kubernetes.io/projected/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-kube-api-access-qvrrc\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:13.951981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.951966    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:13.952085 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.952073    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:13.952177 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.952166    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:13.952296 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.952282    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:13.952470 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.952456    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:13.952574 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.952562    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:13.952707 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:13.952692    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:14.091083 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:14.090996    4889 generic.go:356] "Generic (PLEG): container finished" podID="ffbcb47e-8cd4-4746-b9b2-ff257d8cde01" containerID="37dc7fac3bb494743a71969dbfdb8f9f3fe7782c663a4c20bf85f912070d4f9e" exitCode=0
Oct 30 22:49:14.091083 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:14.091043    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"
Oct 30 22:49:14.091380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:14.091079    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln" event={"ID":"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01","Type":"ContainerDied","Data":"37dc7fac3bb494743a71969dbfdb8f9f3fe7782c663a4c20bf85f912070d4f9e"}
Oct 30 22:49:14.091380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:14.091145    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln" event={"ID":"ffbcb47e-8cd4-4746-b9b2-ff257d8cde01","Type":"ContainerDied","Data":"3cafad3493c80bf9948988d568988df4a24a0c880d3d17a426d2c09b5fa37ad1"}
Oct 30 22:49:14.091879 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:14.091174    4889 scope.go:122] "RemoveContainer" containerID="37dc7fac3bb494743a71969dbfdb8f9f3fe7782c663a4c20bf85f912070d4f9e"
Oct 30 22:49:14.145352 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:14.145324    4889 scope.go:122] "RemoveContainer" containerID="37dc7fac3bb494743a71969dbfdb8f9f3fe7782c663a4c20bf85f912070d4f9e"
Oct 30 22:49:14.145783 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:49:14.145747    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37dc7fac3bb494743a71969dbfdb8f9f3fe7782c663a4c20bf85f912070d4f9e\": container with ID starting with 37dc7fac3bb494743a71969dbfdb8f9f3fe7782c663a4c20bf85f912070d4f9e not found: ID does not exist" containerID="37dc7fac3bb494743a71969dbfdb8f9f3fe7782c663a4c20bf85f912070d4f9e"
Oct 30 22:49:14.145871 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:14.145791    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37dc7fac3bb494743a71969dbfdb8f9f3fe7782c663a4c20bf85f912070d4f9e"} err="failed to get container status \"37dc7fac3bb494743a71969dbfdb8f9f3fe7782c663a4c20bf85f912070d4f9e\": rpc error: code = NotFound desc = could not find container \"37dc7fac3bb494743a71969dbfdb8f9f3fe7782c663a4c20bf85f912070d4f9e\": container with ID starting with 37dc7fac3bb494743a71969dbfdb8f9f3fe7782c663a4c20bf85f912070d4f9e not found: ID does not exist"
Oct 30 22:49:14.164015 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:14.163976    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"]
Oct 30 22:49:14.169927 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:14.169895    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-mthln"]
Oct 30 22:49:14.530740 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:14.530701    4889 patch_prober.go:29] interesting pod/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Oct 30 22:49:14.531098 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:14.531067    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs" podUID="b877606e-a40b-4013-97a3-5087c7b47ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Oct 30 22:49:14.777605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:14.777569    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffbcb47e-8cd4-4746-b9b2-ff257d8cde01" path="/var/lib/kubelet/pods/ffbcb47e-8cd4-4746-b9b2-ff257d8cde01/volumes"
Oct 30 22:49:14.831306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:14.826108    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:15.474505 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:15.474474    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:15.532780 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:15.532738    4889 patch_prober.go:29] interesting pod/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Oct 30 22:49:15.534288 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:15.534252    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs" podUID="b877606e-a40b-4013-97a3-5087c7b47ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Oct 30 22:49:16.148990 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.148950    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"
Oct 30 22:49:16.162939 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.162910    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"
Oct 30 22:49:16.619719 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.616369    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:16.628413 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.628313    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:16.632632 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.632548    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:16.638372 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.638337    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:16.687438 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.687407    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:49:16.699081 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.699049    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fca6fa2-01cd-4f49-96c3-60da13e5c889-kube-api-access-dmdvw\" (UniqueName: \"kubernetes.io/projected/9fca6fa2-01cd-4f49-96c3-60da13e5c889-kube-api-access-dmdvw\") pod \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") "
Oct 30 22:49:16.699298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.699105    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-credential-socket\") pod \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") "
Oct 30 22:49:16.699298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.699132    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-token\" (UniqueName: \"kubernetes.io/projected/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-token\") pod \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") "
Oct 30 22:49:16.699298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.699186    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-podinfo\") pod \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") "
Oct 30 22:49:16.699298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.699215    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-workload-socket\") pod \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") "
Oct 30 22:49:16.699522 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.699345    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-envoy\") pod \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") "
Oct 30 22:49:16.699522 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.699380    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-data\") pod \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") "
Oct 30 22:49:16.699522 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.699471    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-credential-socket" pod "9fca6fa2-01cd-4f49-96c3-60da13e5c889" (UID: "9fca6fa2-01cd-4f49-96c3-60da13e5c889"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:16.699701 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.699536    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istiod-ca-cert\") pod \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") "
Oct 30 22:49:16.699701 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.699604    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-workload-certs\") pod \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\" (UID: \"9fca6fa2-01cd-4f49-96c3-60da13e5c889\") "
Oct 30 22:49:16.699819 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.699705    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-data" pod "9fca6fa2-01cd-4f49-96c3-60da13e5c889" (UID: "9fca6fa2-01cd-4f49-96c3-60da13e5c889"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:16.699819 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.699776    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-workload-socket" pod "9fca6fa2-01cd-4f49-96c3-60da13e5c889" (UID: "9fca6fa2-01cd-4f49-96c3-60da13e5c889"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:16.700541 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.700009    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-workload-certs" pod "9fca6fa2-01cd-4f49-96c3-60da13e5c889" (UID: "9fca6fa2-01cd-4f49-96c3-60da13e5c889"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:16.700541 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.700464    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:16.700541 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.700486    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:16.700541 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.700483    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istiod-ca-cert" pod "9fca6fa2-01cd-4f49-96c3-60da13e5c889" (UID: "9fca6fa2-01cd-4f49-96c3-60da13e5c889"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:16.700541 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.700501    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:16.700541 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.700518    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:16.701810 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.701779    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-token" pod "9fca6fa2-01cd-4f49-96c3-60da13e5c889" (UID: "9fca6fa2-01cd-4f49-96c3-60da13e5c889"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:16.701921 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.701859    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fca6fa2-01cd-4f49-96c3-60da13e5c889-kube-api-access-dmdvw" pod "9fca6fa2-01cd-4f49-96c3-60da13e5c889" (UID: "9fca6fa2-01cd-4f49-96c3-60da13e5c889"). InnerVolumeSpecName "kube-api-access-dmdvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:16.702214 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.702197    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-podinfo" pod "9fca6fa2-01cd-4f49-96c3-60da13e5c889" (UID: "9fca6fa2-01cd-4f49-96c3-60da13e5c889"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:49:16.702414 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.702396    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-envoy" pod "9fca6fa2-01cd-4f49-96c3-60da13e5c889" (UID: "9fca6fa2-01cd-4f49-96c3-60da13e5c889"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:16.804277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.802217    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:16.804277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.802292    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmdvw\" (UniqueName: \"kubernetes.io/projected/9fca6fa2-01cd-4f49-96c3-60da13e5c889-kube-api-access-dmdvw\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:16.804277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.802323    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:16.804277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.802339    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:16.804277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:16.802353    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fca6fa2-01cd-4f49-96c3-60da13e5c889-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:17.117895 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:17.117794    4889 generic.go:356] "Generic (PLEG): container finished" podID="9fca6fa2-01cd-4f49-96c3-60da13e5c889" containerID="73c14c5ce12b23c158a3d4542fce534c4b13e1577f8d541bdffa24311c0883b6" exitCode=0
Oct 30 22:49:17.117895 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:17.117860    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"
Oct 30 22:49:17.118130 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:17.117880    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz" event={"ID":"9fca6fa2-01cd-4f49-96c3-60da13e5c889","Type":"ContainerDied","Data":"73c14c5ce12b23c158a3d4542fce534c4b13e1577f8d541bdffa24311c0883b6"}
Oct 30 22:49:17.118130 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:17.117945    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz" event={"ID":"9fca6fa2-01cd-4f49-96c3-60da13e5c889","Type":"ContainerDied","Data":"6fbfeb461df3144d86128b40ae1f86cc46d4cd355cda6f2ff0e0148f9d4dd9f8"}
Oct 30 22:49:17.118130 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:17.117968    4889 scope.go:122] "RemoveContainer" containerID="73c14c5ce12b23c158a3d4542fce534c4b13e1577f8d541bdffa24311c0883b6"
Oct 30 22:49:17.130839 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:17.130801    4889 scope.go:122] "RemoveContainer" containerID="73c14c5ce12b23c158a3d4542fce534c4b13e1577f8d541bdffa24311c0883b6"
Oct 30 22:49:17.131024 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:17.131007    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"
Oct 30 22:49:17.131209 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:49:17.131187    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73c14c5ce12b23c158a3d4542fce534c4b13e1577f8d541bdffa24311c0883b6\": container with ID starting with 73c14c5ce12b23c158a3d4542fce534c4b13e1577f8d541bdffa24311c0883b6 not found: ID does not exist" containerID="73c14c5ce12b23c158a3d4542fce534c4b13e1577f8d541bdffa24311c0883b6"
Oct 30 22:49:17.131278 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:17.131218    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73c14c5ce12b23c158a3d4542fce534c4b13e1577f8d541bdffa24311c0883b6"} err="failed to get container status \"73c14c5ce12b23c158a3d4542fce534c4b13e1577f8d541bdffa24311c0883b6\": rpc error: code = NotFound desc = could not find container \"73c14c5ce12b23c158a3d4542fce534c4b13e1577f8d541bdffa24311c0883b6\": container with ID starting with 73c14c5ce12b23c158a3d4542fce534c4b13e1577f8d541bdffa24311c0883b6 not found: ID does not exist"
Oct 30 22:49:17.224025 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:17.223986    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"]
Oct 30 22:49:17.236957 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:17.236918    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8c8cz"]
Oct 30 22:49:18.318635 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:18.318592    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:18.776159 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:18.776123    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fca6fa2-01cd-4f49-96c3-60da13e5c889" path="/var/lib/kubelet/pods/9fca6fa2-01cd-4f49-96c3-60da13e5c889/volumes"
Oct 30 22:49:19.392930 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:19.392886    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5"]
Oct 30 22:49:19.427810 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:19.427772    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf"]
Oct 30 22:49:19.588787 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:19.588744    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:19.787125 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:19.787086    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:20.117782 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:20.117672    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:21.791049 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:21.790948    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:22.819948 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:22.819911    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d"]
Oct 30 22:49:23.017712 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:23.017671    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:23.201603 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:23.201507    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:23.369145 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:23.369109    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:23.970133 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:23.969972    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:24.361660 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:24.361610    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:24.394260 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:24.394162    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5" podUID="92be936b-b994-4bb9-ba7c-53845b589684" containerName="payload-processing" containerID="cri-o://c455df5999a73be63a9464c1d9f1d4f878b85f931112c3662985a477e35eb913" gracePeriod=25
Oct 30 22:49:24.428712 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:24.428669    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf" podUID="8e006a58-a7ed-4f12-8c3d-f0db13e46fc2" containerName="payload-pre-processing" containerID="cri-o://88ab58c202110a33de862dc5056bb841302a78e990ef7526e86ad348670cfd6f" gracePeriod=25
Oct 30 22:49:24.537771 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:24.537723    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:24.692926 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:24.692893    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5"
Oct 30 22:49:24.835474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:24.835433    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92be936b-b994-4bb9-ba7c-53845b589684-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92be936b-b994-4bb9-ba7c-53845b589684-plugins-config-volume\") pod \"92be936b-b994-4bb9-ba7c-53845b589684\" (UID: \"92be936b-b994-4bb9-ba7c-53845b589684\") "
Oct 30 22:49:24.835688 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:24.835605    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92be936b-b994-4bb9-ba7c-53845b589684-kube-api-access-twgj2\" (UniqueName: \"kubernetes.io/projected/92be936b-b994-4bb9-ba7c-53845b589684-kube-api-access-twgj2\") pod \"92be936b-b994-4bb9-ba7c-53845b589684\" (UID: \"92be936b-b994-4bb9-ba7c-53845b589684\") "
Oct 30 22:49:24.835865 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:24.835837    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92be936b-b994-4bb9-ba7c-53845b589684-plugins-config-volume" pod "92be936b-b994-4bb9-ba7c-53845b589684" (UID: "92be936b-b994-4bb9-ba7c-53845b589684"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:24.836389 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:24.836355    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92be936b-b994-4bb9-ba7c-53845b589684-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:24.838036 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:24.838017    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92be936b-b994-4bb9-ba7c-53845b589684-kube-api-access-twgj2" pod "92be936b-b994-4bb9-ba7c-53845b589684" (UID: "92be936b-b994-4bb9-ba7c-53845b589684"). InnerVolumeSpecName "kube-api-access-twgj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:24.938211 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:24.938124    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twgj2\" (UniqueName: \"kubernetes.io/projected/92be936b-b994-4bb9-ba7c-53845b589684-kube-api-access-twgj2\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:24.982987 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:24.982949    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:25.219605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:25.219517    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5" event={"ID":"92be936b-b994-4bb9-ba7c-53845b589684","Type":"ContainerDied","Data":"c455df5999a73be63a9464c1d9f1d4f878b85f931112c3662985a477e35eb913"}
Oct 30 22:49:25.219605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:25.219481    4889 generic.go:356] "Generic (PLEG): container finished" podID="92be936b-b994-4bb9-ba7c-53845b589684" containerID="c455df5999a73be63a9464c1d9f1d4f878b85f931112c3662985a477e35eb913" exitCode=0
Oct 30 22:49:25.219895 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:25.219870    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5" event={"ID":"92be936b-b994-4bb9-ba7c-53845b589684","Type":"ContainerDied","Data":"92130609c3e85283d6a438d434df289387a4b72c7b27e6945aac4daf388ff4db"}
Oct 30 22:49:25.219952 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:25.219609    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5"
Oct 30 22:49:25.219952 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:25.219919    4889 scope.go:122] "RemoveContainer" containerID="c455df5999a73be63a9464c1d9f1d4f878b85f931112c3662985a477e35eb913"
Oct 30 22:49:25.232826 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:25.232802    4889 scope.go:122] "RemoveContainer" containerID="c455df5999a73be63a9464c1d9f1d4f878b85f931112c3662985a477e35eb913"
Oct 30 22:49:25.233135 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:49:25.233114    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c455df5999a73be63a9464c1d9f1d4f878b85f931112c3662985a477e35eb913\": container with ID starting with c455df5999a73be63a9464c1d9f1d4f878b85f931112c3662985a477e35eb913 not found: ID does not exist" containerID="c455df5999a73be63a9464c1d9f1d4f878b85f931112c3662985a477e35eb913"
Oct 30 22:49:25.233208 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:25.233145    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c455df5999a73be63a9464c1d9f1d4f878b85f931112c3662985a477e35eb913"} err="failed to get container status \"c455df5999a73be63a9464c1d9f1d4f878b85f931112c3662985a477e35eb913\": rpc error: code = NotFound desc = could not find container \"c455df5999a73be63a9464c1d9f1d4f878b85f931112c3662985a477e35eb913\": container with ID starting with c455df5999a73be63a9464c1d9f1d4f878b85f931112c3662985a477e35eb913 not found: ID does not exist"
Oct 30 22:49:25.280580 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:25.280541    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5"]
Oct 30 22:49:25.291705 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:25.291668    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3acfdb-7f8bb6c86f-7bsb5"]
Oct 30 22:49:25.844807 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:25.844765    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:26.777915 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:26.777875    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92be936b-b994-4bb9-ba7c-53845b589684" path="/var/lib/kubelet/pods/92be936b-b994-4bb9-ba7c-53845b589684/volumes"
Oct 30 22:49:27.550810 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:27.550764    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:27.820564 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:27.820456    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d" podUID="9d8d205a-47b3-45f7-9262-3083ea98e5c9" containerName="payload-processing" containerID="cri-o://225e39ef4eeeca381311b5a437273cca675472a01372f933060111d7367cda90" gracePeriod=25
Oct 30 22:49:29.320719 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:29.320681    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:30.197021 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:30.196978    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf" podUID="8e006a58-a7ed-4f12-8c3d-f0db13e46fc2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:49:31.019527 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:31.019479    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:31.342886 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:31.342749    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:31.511903 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:31.511847    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:31.580567 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:31.580528    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"]
Oct 30 22:49:31.580852 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:31.580821    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs" podUID="b877606e-a40b-4013-97a3-5087c7b47ce6" containerName="istio-proxy" containerID="cri-o://42454bc96dc4435253654aff08064138cb9daabfa54173de405b8080e9dd634d" gracePeriod=30
Oct 30 22:49:31.620388 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:31.620291    4889 patch_prober.go:29] interesting pod/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 30 22:49:31.620388 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:31.620351    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs" podUID="b877606e-a40b-4013-97a3-5087c7b47ce6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:49:31.685482 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:31.685445    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:31.888204 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:31.888120    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d" podUID="9d8d205a-47b3-45f7-9262-3083ea98e5c9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:49:32.804558 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:32.804508    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:34.735042 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:34.735003    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:36.570277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:36.570201    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:36.874627 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:36.874600    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:36.952365 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:36.952326    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-workload-certs\") pod \"b877606e-a40b-4013-97a3-5087c7b47ce6\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") "
Oct 30 22:49:36.952598 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:36.952393    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-workload-socket\") pod \"b877606e-a40b-4013-97a3-5087c7b47ce6\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") "
Oct 30 22:49:36.952598 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:36.952499    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-credential-socket\") pod \"b877606e-a40b-4013-97a3-5087c7b47ce6\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") "
Oct 30 22:49:36.952598 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:36.952540    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-data\") pod \"b877606e-a40b-4013-97a3-5087c7b47ce6\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") "
Oct 30 22:49:36.952598 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:36.952586    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-envoy\") pod \"b877606e-a40b-4013-97a3-5087c7b47ce6\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") "
Oct 30 22:49:36.952884 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:36.952672    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-token\" (UniqueName: \"kubernetes.io/projected/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-token\") pod \"b877606e-a40b-4013-97a3-5087c7b47ce6\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") "
Oct 30 22:49:36.952884 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:36.952711    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b877606e-a40b-4013-97a3-5087c7b47ce6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b877606e-a40b-4013-97a3-5087c7b47ce6-istiod-ca-cert\") pod \"b877606e-a40b-4013-97a3-5087c7b47ce6\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") "
Oct 30 22:49:36.952884 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:36.952799    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-podinfo\") pod \"b877606e-a40b-4013-97a3-5087c7b47ce6\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") "
Oct 30 22:49:36.953042 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:36.952943    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b877606e-a40b-4013-97a3-5087c7b47ce6-kube-api-access-d85wn\" (UniqueName: \"kubernetes.io/projected/b877606e-a40b-4013-97a3-5087c7b47ce6-kube-api-access-d85wn\") pod \"b877606e-a40b-4013-97a3-5087c7b47ce6\" (UID: \"b877606e-a40b-4013-97a3-5087c7b47ce6\") "
Oct 30 22:49:36.953474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:36.953445    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-data" pod "b877606e-a40b-4013-97a3-5087c7b47ce6" (UID: "b877606e-a40b-4013-97a3-5087c7b47ce6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:36.953643 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:36.953459    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-workload-socket" pod "b877606e-a40b-4013-97a3-5087c7b47ce6" (UID: "b877606e-a40b-4013-97a3-5087c7b47ce6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:36.954314 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:36.954163    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b877606e-a40b-4013-97a3-5087c7b47ce6-istiod-ca-cert" pod "b877606e-a40b-4013-97a3-5087c7b47ce6" (UID: "b877606e-a40b-4013-97a3-5087c7b47ce6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:36.954314 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:36.954273    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-workload-certs" pod "b877606e-a40b-4013-97a3-5087c7b47ce6" (UID: "b877606e-a40b-4013-97a3-5087c7b47ce6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:36.955103 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:36.954712    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:36.955103 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:36.954732    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:36.955103 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:36.954745    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:36.955103 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:36.954760    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b877606e-a40b-4013-97a3-5087c7b47ce6-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:36.955103 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:36.954995    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-credential-socket" pod "b877606e-a40b-4013-97a3-5087c7b47ce6" (UID: "b877606e-a40b-4013-97a3-5087c7b47ce6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:36.957630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:36.957593    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-envoy" pod "b877606e-a40b-4013-97a3-5087c7b47ce6" (UID: "b877606e-a40b-4013-97a3-5087c7b47ce6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:36.957757 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:36.957738    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b877606e-a40b-4013-97a3-5087c7b47ce6-kube-api-access-d85wn" pod "b877606e-a40b-4013-97a3-5087c7b47ce6" (UID: "b877606e-a40b-4013-97a3-5087c7b47ce6"). InnerVolumeSpecName "kube-api-access-d85wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:36.959530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:36.958517    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-podinfo" pod "b877606e-a40b-4013-97a3-5087c7b47ce6" (UID: "b877606e-a40b-4013-97a3-5087c7b47ce6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:49:36.959530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:36.958806    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-token" pod "b877606e-a40b-4013-97a3-5087c7b47ce6" (UID: "b877606e-a40b-4013-97a3-5087c7b47ce6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:37.056298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:37.056263    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:37.056298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:37.056297    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d85wn\" (UniqueName: \"kubernetes.io/projected/b877606e-a40b-4013-97a3-5087c7b47ce6-kube-api-access-d85wn\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:37.056298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:37.056312    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:37.056563 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:37.056323    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:37.056563 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:37.056335    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b877606e-a40b-4013-97a3-5087c7b47ce6-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:37.357472 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:37.357434    4889 generic.go:356] "Generic (PLEG): container finished" podID="b877606e-a40b-4013-97a3-5087c7b47ce6" containerID="42454bc96dc4435253654aff08064138cb9daabfa54173de405b8080e9dd634d" exitCode=0
Oct 30 22:49:37.357677 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:37.357494    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"
Oct 30 22:49:37.357677 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:37.357530    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs" event={"ID":"b877606e-a40b-4013-97a3-5087c7b47ce6","Type":"ContainerDied","Data":"42454bc96dc4435253654aff08064138cb9daabfa54173de405b8080e9dd634d"}
Oct 30 22:49:37.357677 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:37.357593    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs" event={"ID":"b877606e-a40b-4013-97a3-5087c7b47ce6","Type":"ContainerDied","Data":"d55f464344ba7b249bcac4bf1f0a66294d5595d95e23eb02b153bad6699dc15b"}
Oct 30 22:49:37.357677 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:37.357620    4889 scope.go:122] "RemoveContainer" containerID="42454bc96dc4435253654aff08064138cb9daabfa54173de405b8080e9dd634d"
Oct 30 22:49:37.370016 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:37.369988    4889 scope.go:122] "RemoveContainer" containerID="42454bc96dc4435253654aff08064138cb9daabfa54173de405b8080e9dd634d"
Oct 30 22:49:37.370390 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:49:37.370367    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42454bc96dc4435253654aff08064138cb9daabfa54173de405b8080e9dd634d\": container with ID starting with 42454bc96dc4435253654aff08064138cb9daabfa54173de405b8080e9dd634d not found: ID does not exist" containerID="42454bc96dc4435253654aff08064138cb9daabfa54173de405b8080e9dd634d"
Oct 30 22:49:37.370466 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:37.370400    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42454bc96dc4435253654aff08064138cb9daabfa54173de405b8080e9dd634d"} err="failed to get container status \"42454bc96dc4435253654aff08064138cb9daabfa54173de405b8080e9dd634d\": rpc error: code = NotFound desc = could not find container \"42454bc96dc4435253654aff08064138cb9daabfa54173de405b8080e9dd634d\": container with ID starting with 42454bc96dc4435253654aff08064138cb9daabfa54173de405b8080e9dd634d not found: ID does not exist"
Oct 30 22:49:37.415773 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:37.415738    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"]
Oct 30 22:49:37.430264 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:37.430212    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-p62qs"]
Oct 30 22:49:38.350569 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:38.350536    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:38.776356 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:38.776320    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b877606e-a40b-4013-97a3-5087c7b47ce6" path="/var/lib/kubelet/pods/b877606e-a40b-4013-97a3-5087c7b47ce6/volumes"
Oct 30 22:49:40.139471 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:40.139427    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:40.196909 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:40.196866    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf" podUID="8e006a58-a7ed-4f12-8c3d-f0db13e46fc2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:49:40.304033 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:49:40.303995    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:49:41.887946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:41.887897    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d" podUID="9d8d205a-47b3-45f7-9262-3083ea98e5c9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:49:42.016920 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.016885    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:42.547187 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.547147    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"]
Oct 30 22:49:42.547913 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.547898    4889 cpu_manager.go:414] "containerMap: removing container" podUID="92be936b-b994-4bb9-ba7c-53845b589684" containerName="payload-processing"
Oct 30 22:49:42.547962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.547916    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92be936b-b994-4bb9-ba7c-53845b589684" containerName="payload-processing"
Oct 30 22:49:42.547962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.547930    4889 cpu_manager.go:414] "containerMap: removing container" podUID="b877606e-a40b-4013-97a3-5087c7b47ce6" containerName="istio-proxy"
Oct 30 22:49:42.547962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.547936    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b877606e-a40b-4013-97a3-5087c7b47ce6" containerName="istio-proxy"
Oct 30 22:49:42.547962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.547950    4889 cpu_manager.go:414] "containerMap: removing container" podUID="9fca6fa2-01cd-4f49-96c3-60da13e5c889" containerName="istio-proxy"
Oct 30 22:49:42.547962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.547955    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fca6fa2-01cd-4f49-96c3-60da13e5c889" containerName="istio-proxy"
Oct 30 22:49:42.547962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.547963    4889 cpu_manager.go:414] "containerMap: removing container" podUID="ffbcb47e-8cd4-4746-b9b2-ff257d8cde01" containerName="istio-proxy"
Oct 30 22:49:42.548157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.547968    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffbcb47e-8cd4-4746-b9b2-ff257d8cde01" containerName="istio-proxy"
Oct 30 22:49:42.548157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.548094    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="92be936b-b994-4bb9-ba7c-53845b589684" containerName="payload-processing"
Oct 30 22:49:42.548157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.548102    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="b877606e-a40b-4013-97a3-5087c7b47ce6" containerName="istio-proxy"
Oct 30 22:49:42.548157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.548110    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fca6fa2-01cd-4f49-96c3-60da13e5c889" containerName="istio-proxy"
Oct 30 22:49:42.548157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.548117    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffbcb47e-8cd4-4746-b9b2-ff257d8cde01" containerName="istio-proxy"
Oct 30 22:49:42.553748 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.553725    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.574096 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.574059    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"]
Oct 30 22:49:42.583355 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.583323    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-6f2eba-openshift-default-dockercfg-kkm8p\""
Oct 30 22:49:42.590200 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.590159    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv"]
Oct 30 22:49:42.639059 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.639020    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istiod-ca-cert\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.639271 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.639091    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-podinfo\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.639271 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.639180    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-token\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.639271 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.639247    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-data\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.639431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.639278    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-envoy\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.639431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.639408    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-credential-socket\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.639545 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.639528    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwb4z\" (UniqueName: \"kubernetes.io/projected/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-kube-api-access-pwb4z\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.639615 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.639566    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-workload-socket\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.639615 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.639598    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-workload-certs\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.741370 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.741331    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istiod-ca-cert\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.741599 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.741398    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-podinfo\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.741599 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.741435    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-token\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.741599 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.741469    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-data\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.741599 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.741499    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-envoy\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.741859 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.741617    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-credential-socket\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.741859 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.741758    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwb4z\" (UniqueName: \"kubernetes.io/projected/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-kube-api-access-pwb4z\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.741859 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.741806    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-workload-socket\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.741859 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.741834    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-workload-certs\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.742087 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.741988    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-data\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.742087 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.742043    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-credential-socket\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.742202 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.742169    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istiod-ca-cert\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.742291 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.742243    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-workload-certs\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.742411 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.742392    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-workload-socket\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.744337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.744313    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-podinfo\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.744452 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.744336    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-envoy\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.763842 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.763804    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-token\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.764521 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.764496    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwb4z\" (UniqueName: \"kubernetes.io/projected/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-kube-api-access-pwb4z\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.782853 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.778519    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:42.870073 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.869981    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:42.949377 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:42.949332    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:43.221516 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:43.221417    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"]
Oct 30 22:49:43.228315 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:49:43.228279    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb2ce23b_fd13_45c7_8e1d_f9be28fc05d4.slice/crio-76d624fb03dd61f8eda18203d86b3b7b822d18a9055e933398799e4ef3149398 WatchSource:0}: Error finding container 76d624fb03dd61f8eda18203d86b3b7b822d18a9055e933398799e4ef3149398: Status 404 returned error can't find the container with id 76d624fb03dd61f8eda18203d86b3b7b822d18a9055e933398799e4ef3149398
Oct 30 22:49:43.245218 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:43.245173    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:49:43.245505 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:43.245483    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:49:43.245633 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:43.245619    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:49:43.408834 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:43.408785    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d" event={"ID":"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4","Type":"ContainerStarted","Data":"69e6b71e4c80c0aa052106f1e0649b85eee418eefe625fcfc8d37c1716a080f7"}
Oct 30 22:49:43.409035 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:43.408850    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d" event={"ID":"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4","Type":"ContainerStarted","Data":"76d624fb03dd61f8eda18203d86b3b7b822d18a9055e933398799e4ef3149398"}
Oct 30 22:49:43.459072 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:43.459004    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d" podStartSLOduration=1.458984106 podStartE2EDuration="1.458984106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:43.455106559 +0000 UTC m=+2568.811473077" watchObservedRunningTime="2026-10-30 22:49:43.458984106 +0000 UTC m=+2568.815350423"
Oct 30 22:49:43.814496 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:43.814460    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:43.870997 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:43.870960    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:44.841850 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:44.841823    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf"
Oct 30 22:49:44.872308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:44.872259    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Oct 30 22:49:44.872501 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:44.872330    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d" podUID="fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Oct 30 22:49:44.981719 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:44.981632    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e006a58-a7ed-4f12-8c3d-f0db13e46fc2-kube-api-access-k2mwn\" (UniqueName: \"kubernetes.io/projected/8e006a58-a7ed-4f12-8c3d-f0db13e46fc2-kube-api-access-k2mwn\") pod \"8e006a58-a7ed-4f12-8c3d-f0db13e46fc2\" (UID: \"8e006a58-a7ed-4f12-8c3d-f0db13e46fc2\") "
Oct 30 22:49:44.981901 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:44.981780    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e006a58-a7ed-4f12-8c3d-f0db13e46fc2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e006a58-a7ed-4f12-8c3d-f0db13e46fc2-plugins-config-volume\") pod \"8e006a58-a7ed-4f12-8c3d-f0db13e46fc2\" (UID: \"8e006a58-a7ed-4f12-8c3d-f0db13e46fc2\") "
Oct 30 22:49:44.982235 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:44.982196    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e006a58-a7ed-4f12-8c3d-f0db13e46fc2-plugins-config-volume" pod "8e006a58-a7ed-4f12-8c3d-f0db13e46fc2" (UID: "8e006a58-a7ed-4f12-8c3d-f0db13e46fc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:44.982563 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:44.982544    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e006a58-a7ed-4f12-8c3d-f0db13e46fc2-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:44.984077 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:44.984052    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e006a58-a7ed-4f12-8c3d-f0db13e46fc2-kube-api-access-k2mwn" pod "8e006a58-a7ed-4f12-8c3d-f0db13e46fc2" (UID: "8e006a58-a7ed-4f12-8c3d-f0db13e46fc2"). InnerVolumeSpecName "kube-api-access-k2mwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:45.084007 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:45.083965    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2mwn\" (UniqueName: \"kubernetes.io/projected/8e006a58-a7ed-4f12-8c3d-f0db13e46fc2-kube-api-access-k2mwn\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:45.357127 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:45.357087    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:45.451375 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:45.451342    4889 generic.go:356] "Generic (PLEG): container finished" podID="8e006a58-a7ed-4f12-8c3d-f0db13e46fc2" containerID="88ab58c202110a33de862dc5056bb841302a78e990ef7526e86ad348670cfd6f" exitCode=0
Oct 30 22:49:45.451562 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:45.451391    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf"
Oct 30 22:49:45.451562 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:45.451422    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf" event={"ID":"8e006a58-a7ed-4f12-8c3d-f0db13e46fc2","Type":"ContainerDied","Data":"88ab58c202110a33de862dc5056bb841302a78e990ef7526e86ad348670cfd6f"}
Oct 30 22:49:45.451562 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:45.451473    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf" event={"ID":"8e006a58-a7ed-4f12-8c3d-f0db13e46fc2","Type":"ContainerDied","Data":"b0517a05a4a74a90136ede69f86864da70a096b840673c14f69602908aed4091"}
Oct 30 22:49:45.451562 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:45.451498    4889 scope.go:122] "RemoveContainer" containerID="88ab58c202110a33de862dc5056bb841302a78e990ef7526e86ad348670cfd6f"
Oct 30 22:49:45.464923 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:45.464902    4889 scope.go:122] "RemoveContainer" containerID="88ab58c202110a33de862dc5056bb841302a78e990ef7526e86ad348670cfd6f"
Oct 30 22:49:45.465341 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:49:45.465315    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88ab58c202110a33de862dc5056bb841302a78e990ef7526e86ad348670cfd6f\": container with ID starting with 88ab58c202110a33de862dc5056bb841302a78e990ef7526e86ad348670cfd6f not found: ID does not exist" containerID="88ab58c202110a33de862dc5056bb841302a78e990ef7526e86ad348670cfd6f"
Oct 30 22:49:45.465407 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:45.465354    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ab58c202110a33de862dc5056bb841302a78e990ef7526e86ad348670cfd6f"} err="failed to get container status \"88ab58c202110a33de862dc5056bb841302a78e990ef7526e86ad348670cfd6f\": rpc error: code = NotFound desc = could not find container \"88ab58c202110a33de862dc5056bb841302a78e990ef7526e86ad348670cfd6f\": container with ID starting with 88ab58c202110a33de862dc5056bb841302a78e990ef7526e86ad348670cfd6f not found: ID does not exist"
Oct 30 22:49:45.531173 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:45.531135    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf"]
Oct 30 22:49:45.539260 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:45.534607    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:45.549251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:45.541557    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3acfdb-66c888cfdc-ktrkf"]
Oct 30 22:49:45.836715 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:45.836643    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:45.871165 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:45.871126    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Oct 30 22:49:45.871649 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:45.871180    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d" podUID="fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Oct 30 22:49:46.035359 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:46.035320    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:46.246730 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:46.230934    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:46.407335 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:46.406088    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:46.594252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:46.593038    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:46.779182 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:46.779148    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e006a58-a7ed-4f12-8c3d-f0db13e46fc2" path="/var/lib/kubelet/pods/8e006a58-a7ed-4f12-8c3d-f0db13e46fc2/volumes"
Oct 30 22:49:46.836964 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:46.836923    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:46.871244 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:46.871151    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Oct 30 22:49:46.871244 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:46.871201    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d" podUID="fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Oct 30 22:49:47.590506 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:47.590457    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv" podUID="8ceb7603-9c15-4c92-9dbd-bd6c45828ef8" containerName="payload-processing" containerID="cri-o://20a81b12ccb2708cb2029fbd1674c2150741ed20d5a46bbbc0b72705a9a9efd0" gracePeriod=25
Oct 30 22:49:47.836109 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:47.836057    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv" podUID="8ceb7603-9c15-4c92-9dbd-bd6c45828ef8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:49:47.871437 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:47.871385    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Oct 30 22:49:47.871904 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:47.871460    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d" podUID="fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Oct 30 22:49:48.168073 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:48.168045    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d"
Oct 30 22:49:48.248259 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:48.248178    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:48.264694 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:48.257376    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8d205a-47b3-45f7-9262-3083ea98e5c9-plugins-config-volume" pod "9d8d205a-47b3-45f7-9262-3083ea98e5c9" (UID: "9d8d205a-47b3-45f7-9262-3083ea98e5c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:48.264694 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:48.256599    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d8d205a-47b3-45f7-9262-3083ea98e5c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d8d205a-47b3-45f7-9262-3083ea98e5c9-plugins-config-volume\") pod \"9d8d205a-47b3-45f7-9262-3083ea98e5c9\" (UID: \"9d8d205a-47b3-45f7-9262-3083ea98e5c9\") "
Oct 30 22:49:48.264694 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:48.257742    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d8d205a-47b3-45f7-9262-3083ea98e5c9-kube-api-access-dxtcm\" (UniqueName: \"kubernetes.io/projected/9d8d205a-47b3-45f7-9262-3083ea98e5c9-kube-api-access-dxtcm\") pod \"9d8d205a-47b3-45f7-9262-3083ea98e5c9\" (UID: \"9d8d205a-47b3-45f7-9262-3083ea98e5c9\") "
Oct 30 22:49:48.264694 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:48.259906    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d8d205a-47b3-45f7-9262-3083ea98e5c9-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:48.274480 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:48.274436    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d8d205a-47b3-45f7-9262-3083ea98e5c9-kube-api-access-dxtcm" pod "9d8d205a-47b3-45f7-9262-3083ea98e5c9" (UID: "9d8d205a-47b3-45f7-9262-3083ea98e5c9"). InnerVolumeSpecName "kube-api-access-dxtcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:48.362374 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:48.362283    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxtcm\" (UniqueName: \"kubernetes.io/projected/9d8d205a-47b3-45f7-9262-3083ea98e5c9-kube-api-access-dxtcm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:48.489034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:48.488994    4889 generic.go:356] "Generic (PLEG): container finished" podID="9d8d205a-47b3-45f7-9262-3083ea98e5c9" containerID="225e39ef4eeeca381311b5a437273cca675472a01372f933060111d7367cda90" exitCode=0
Oct 30 22:49:48.489283 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:48.489048    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d"
Oct 30 22:49:48.489283 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:48.489082    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d" event={"ID":"9d8d205a-47b3-45f7-9262-3083ea98e5c9","Type":"ContainerDied","Data":"225e39ef4eeeca381311b5a437273cca675472a01372f933060111d7367cda90"}
Oct 30 22:49:48.489283 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:48.489167    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d" event={"ID":"9d8d205a-47b3-45f7-9262-3083ea98e5c9","Type":"ContainerDied","Data":"6e4b668f517f40084e80023b8b33b9b4279b54afbf6100dd710ca42a00242836"}
Oct 30 22:49:48.489283 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:48.489203    4889 scope.go:122] "RemoveContainer" containerID="225e39ef4eeeca381311b5a437273cca675472a01372f933060111d7367cda90"
Oct 30 22:49:48.501411 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:48.501389    4889 scope.go:122] "RemoveContainer" containerID="225e39ef4eeeca381311b5a437273cca675472a01372f933060111d7367cda90"
Oct 30 22:49:48.501799 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:49:48.501760    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"225e39ef4eeeca381311b5a437273cca675472a01372f933060111d7367cda90\": container with ID starting with 225e39ef4eeeca381311b5a437273cca675472a01372f933060111d7367cda90 not found: ID does not exist" containerID="225e39ef4eeeca381311b5a437273cca675472a01372f933060111d7367cda90"
Oct 30 22:49:48.501870 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:48.501812    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225e39ef4eeeca381311b5a437273cca675472a01372f933060111d7367cda90"} err="failed to get container status \"225e39ef4eeeca381311b5a437273cca675472a01372f933060111d7367cda90\": rpc error: code = NotFound desc = could not find container \"225e39ef4eeeca381311b5a437273cca675472a01372f933060111d7367cda90\": container with ID starting with 225e39ef4eeeca381311b5a437273cca675472a01372f933060111d7367cda90 not found: ID does not exist"
Oct 30 22:49:48.560702 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:48.560667    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d"]
Oct 30 22:49:48.564701 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:48.564666    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bc48cc-7d467bc759-tqr4d"]
Oct 30 22:49:48.776370 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:48.776328    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d8d205a-47b3-45f7-9262-3083ea98e5c9" path="/var/lib/kubelet/pods/9d8d205a-47b3-45f7-9262-3083ea98e5c9/volumes"
Oct 30 22:49:48.870444 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:48.870406    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Oct 30 22:49:48.870635 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:48.870478    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d" podUID="fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Oct 30 22:49:49.870620 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:49.870581    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Oct 30 22:49:49.871087 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:49.870648    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d" podUID="fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Oct 30 22:49:49.976316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:49.976280    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:50.224304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:50.223918    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:50.418699 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:50.418660    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:50.876797 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:50.876764    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:50.877265 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:50.877249    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:50.878358 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:50.878334    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 22:49:51.923078 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:51.923038    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:52.093737 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:52.093699    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:52.998410 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:52.998370    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"]
Oct 30 22:49:52.999417 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:52.999395    4889 cpu_manager.go:414] "containerMap: removing container" podUID="9d8d205a-47b3-45f7-9262-3083ea98e5c9" containerName="payload-processing"
Oct 30 22:49:52.999417 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:52.999416    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d8d205a-47b3-45f7-9262-3083ea98e5c9" containerName="payload-processing"
Oct 30 22:49:52.999566 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:52.999473    4889 cpu_manager.go:414] "containerMap: removing container" podUID="8e006a58-a7ed-4f12-8c3d-f0db13e46fc2" containerName="payload-pre-processing"
Oct 30 22:49:52.999566 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:52.999479    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e006a58-a7ed-4f12-8c3d-f0db13e46fc2" containerName="payload-pre-processing"
Oct 30 22:49:52.999697 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:52.999580    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d8d205a-47b3-45f7-9262-3083ea98e5c9" containerName="payload-processing"
Oct 30 22:49:52.999697 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:52.999599    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e006a58-a7ed-4f12-8c3d-f0db13e46fc2" containerName="payload-pre-processing"
Oct 30 22:49:53.004130 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.004098    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"
Oct 30 22:49:53.021910 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.021866    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"]
Oct 30 22:49:53.028418 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.028386    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-510f80\"/\"kube-root-ca.crt\""
Oct 30 22:49:53.028582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.028472    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-510f80\"/\"openshift-service-ca.crt\""
Oct 30 22:49:53.029299 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.029278    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-510f80\"/\"default-dockercfg-vlhjl\""
Oct 30 22:49:53.037524 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.037495    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-510f80\"/\"auth-test-model-510f80-kserve-self-signed-certs\""
Oct 30 22:49:53.103916 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.103874    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:53.159420 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.159383    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/818acd09-3afc-4d37-8443-cb3e4e45550b-tls-certs\") pod \"auth-test-model-510f80-kserve-67d6f5b849-5g7km\" (UID: \"818acd09-3afc-4d37-8443-cb3e4e45550b\") " pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"
Oct 30 22:49:53.159641 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.159451    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-model-cache\") pod \"auth-test-model-510f80-kserve-67d6f5b849-5g7km\" (UID: \"818acd09-3afc-4d37-8443-cb3e4e45550b\") " pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"
Oct 30 22:49:53.159641 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.159517    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-home\") pod \"auth-test-model-510f80-kserve-67d6f5b849-5g7km\" (UID: \"818acd09-3afc-4d37-8443-cb3e4e45550b\") " pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"
Oct 30 22:49:53.159641 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.159626    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-dshm\") pod \"auth-test-model-510f80-kserve-67d6f5b849-5g7km\" (UID: \"818acd09-3afc-4d37-8443-cb3e4e45550b\") " pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"
Oct 30 22:49:53.159808 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.159689    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkttb\" (UniqueName: \"kubernetes.io/projected/818acd09-3afc-4d37-8443-cb3e4e45550b-kube-api-access-qkttb\") pod \"auth-test-model-510f80-kserve-67d6f5b849-5g7km\" (UID: \"818acd09-3afc-4d37-8443-cb3e4e45550b\") " pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"
Oct 30 22:49:53.159876 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.159848    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-tmp-dir\") pod \"auth-test-model-510f80-kserve-67d6f5b849-5g7km\" (UID: \"818acd09-3afc-4d37-8443-cb3e4e45550b\") " pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"
Oct 30 22:49:53.264301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.261325    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-tmp-dir\") pod \"auth-test-model-510f80-kserve-67d6f5b849-5g7km\" (UID: \"818acd09-3afc-4d37-8443-cb3e4e45550b\") " pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"
Oct 30 22:49:53.264301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.261451    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/818acd09-3afc-4d37-8443-cb3e4e45550b-tls-certs\") pod \"auth-test-model-510f80-kserve-67d6f5b849-5g7km\" (UID: \"818acd09-3afc-4d37-8443-cb3e4e45550b\") " pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"
Oct 30 22:49:53.264301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.261554    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-model-cache\") pod \"auth-test-model-510f80-kserve-67d6f5b849-5g7km\" (UID: \"818acd09-3afc-4d37-8443-cb3e4e45550b\") " pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"
Oct 30 22:49:53.264301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.261581    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-home\") pod \"auth-test-model-510f80-kserve-67d6f5b849-5g7km\" (UID: \"818acd09-3afc-4d37-8443-cb3e4e45550b\") " pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"
Oct 30 22:49:53.264301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.261631    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-dshm\") pod \"auth-test-model-510f80-kserve-67d6f5b849-5g7km\" (UID: \"818acd09-3afc-4d37-8443-cb3e4e45550b\") " pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"
Oct 30 22:49:53.264301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.261698    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkttb\" (UniqueName: \"kubernetes.io/projected/818acd09-3afc-4d37-8443-cb3e4e45550b-kube-api-access-qkttb\") pod \"auth-test-model-510f80-kserve-67d6f5b849-5g7km\" (UID: \"818acd09-3afc-4d37-8443-cb3e4e45550b\") " pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"
Oct 30 22:49:53.264301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.262757    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-model-cache\") pod \"auth-test-model-510f80-kserve-67d6f5b849-5g7km\" (UID: \"818acd09-3afc-4d37-8443-cb3e4e45550b\") " pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"
Oct 30 22:49:53.264301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.263072    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-tmp-dir\") pod \"auth-test-model-510f80-kserve-67d6f5b849-5g7km\" (UID: \"818acd09-3afc-4d37-8443-cb3e4e45550b\") " pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"
Oct 30 22:49:53.264301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.263410    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-home\") pod \"auth-test-model-510f80-kserve-67d6f5b849-5g7km\" (UID: \"818acd09-3afc-4d37-8443-cb3e4e45550b\") " pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"
Oct 30 22:49:53.266899 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.266865    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/818acd09-3afc-4d37-8443-cb3e4e45550b-tls-certs\") pod \"auth-test-model-510f80-kserve-67d6f5b849-5g7km\" (UID: \"818acd09-3afc-4d37-8443-cb3e4e45550b\") " pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"
Oct 30 22:49:53.277616 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.277577    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-dshm\") pod \"auth-test-model-510f80-kserve-67d6f5b849-5g7km\" (UID: \"818acd09-3afc-4d37-8443-cb3e4e45550b\") " pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"
Oct 30 22:49:53.288960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.288925    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkttb\" (UniqueName: \"kubernetes.io/projected/818acd09-3afc-4d37-8443-cb3e4e45550b-kube-api-access-qkttb\") pod \"auth-test-model-510f80-kserve-67d6f5b849-5g7km\" (UID: \"818acd09-3afc-4d37-8443-cb3e4e45550b\") " pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"
Oct 30 22:49:53.325948 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.325909    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"
Oct 30 22:49:53.643089 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.643053    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"]
Oct 30 22:49:53.643523 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:49:53.643495    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod818acd09_3afc_4d37_8443_cb3e4e45550b.slice/crio-9061686167f0b525c36d4b87fdb4e12db6b574d4fc46282d6aded039956dd085 WatchSource:0}: Error finding container 9061686167f0b525c36d4b87fdb4e12db6b574d4fc46282d6aded039956dd085: Status 404 returned error can't find the container with id 9061686167f0b525c36d4b87fdb4e12db6b574d4fc46282d6aded039956dd085
Oct 30 22:49:53.832369 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:53.832329    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:54.069754 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:54.069712    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:54.239627 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:54.239584    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:54.301277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:54.301208    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"]
Oct 30 22:49:54.301480 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:54.301452    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s" podUID="23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8" containerName="istio-proxy" containerID="cri-o://e8c2089231571ba84c3f9802a2814d7d3c1902509eea0e82f6b9d3e07722e922" gracePeriod=30
Oct 30 22:49:54.425307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:54.404098    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:54.550320 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:54.550273    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km" event={"ID":"818acd09-3afc-4d37-8443-cb3e4e45550b","Type":"ContainerStarted","Data":"e94ecd6016f4cf70d4660f75b443ca4cc111c67d5316d0a3b9dd6bc3532d7132"}
Oct 30 22:49:54.550515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:54.550331    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km" event={"ID":"818acd09-3afc-4d37-8443-cb3e4e45550b","Type":"ContainerStarted","Data":"9061686167f0b525c36d4b87fdb4e12db6b574d4fc46282d6aded039956dd085"}
Oct 30 22:49:54.594338 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:54.594279    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km" podStartSLOduration=2.594264295 podStartE2EDuration="2.594264295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:54.592732788 +0000 UTC m=+2579.949099104" watchObservedRunningTime="2026-10-30 22:49:54.594264295 +0000 UTC m=+2579.950630600"
Oct 30 22:49:55.202388 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:55.202352    4889 patch_prober.go:29] interesting pod/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:49:55.202810 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:55.202418    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s" podUID="23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:49:55.326444 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:55.326402    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"
Oct 30 22:49:55.632502 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:55.632468    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:57.466709 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:57.466669    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:57.828528 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:57.828483    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv" podUID="8ceb7603-9c15-4c92-9dbd-bd6c45828ef8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:49:59.117064 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.117024    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:59.592313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.592280    4889 generic.go:356] "Generic (PLEG): container finished" podID="23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8" containerID="e8c2089231571ba84c3f9802a2814d7d3c1902509eea0e82f6b9d3e07722e922" exitCode=0
Oct 30 22:49:59.592468 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.592352    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s" event={"ID":"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8","Type":"ContainerDied","Data":"e8c2089231571ba84c3f9802a2814d7d3c1902509eea0e82f6b9d3e07722e922"}
Oct 30 22:49:59.592468 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.592409    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s" event={"ID":"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8","Type":"ContainerDied","Data":"08d86dd84f7152bb337be1f701771c320828066ae704cd6667d7c20f47001832"}
Oct 30 22:49:59.592468 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.592429    4889 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08d86dd84f7152bb337be1f701771c320828066ae704cd6667d7c20f47001832"
Oct 30 22:49:59.615261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.615198    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:49:59.756998 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.756944    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-envoy\") pod \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") "
Oct 30 22:49:59.757299 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.757018    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-token\" (UniqueName: \"kubernetes.io/projected/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-token\") pod \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") "
Oct 30 22:49:59.757299 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.757106    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-data\") pod \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") "
Oct 30 22:49:59.757299 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.757142    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-workload-certs\") pod \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") "
Oct 30 22:49:59.757299 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.757177    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-podinfo\") pod \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") "
Oct 30 22:49:59.757299 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.757215    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istiod-ca-cert\") pod \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") "
Oct 30 22:49:59.757299 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.757281    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-kube-api-access-2dcxz\" (UniqueName: \"kubernetes.io/projected/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-kube-api-access-2dcxz\") pod \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") "
Oct 30 22:49:59.757717 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.757393    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-credential-socket\") pod \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") "
Oct 30 22:49:59.757717 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.757491    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-workload-socket\") pod \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\" (UID: \"23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8\") "
Oct 30 22:49:59.757717 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.757547    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-data" pod "23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8" (UID: "23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:59.757717 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.757567    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-workload-certs" pod "23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8" (UID: "23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:59.757717 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.757666    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istiod-ca-cert" pod "23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8" (UID: "23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:59.757980 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.757805    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-credential-socket" pod "23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8" (UID: "23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:59.758034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.757968    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-workload-socket" pod "23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8" (UID: "23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:59.758469 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.758443    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:59.758469 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.758470    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:59.758662 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.758486    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:59.758662 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.758501    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:59.758662 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.758516    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:59.759923 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.759891    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-podinfo" pod "23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8" (UID: "23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:49:59.760029 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.759957    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-token" pod "23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8" (UID: "23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:59.760029 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.759972    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-kube-api-access-2dcxz" pod "23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8" (UID: "23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8"). InnerVolumeSpecName "kube-api-access-2dcxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:59.760572 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.760551    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-envoy" pod "23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8" (UID: "23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:59.859878 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.859839    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:59.859878 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.859876    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:59.859878 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.859884    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:49:59.860134 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:49:59.859893    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dcxz\" (UniqueName: \"kubernetes.io/projected/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8-kube-api-access-2dcxz\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:50:00.599545 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:00.599505    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"
Oct 30 22:50:00.671938 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:00.671908    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"]
Oct 30 22:50:00.685215 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:00.685182    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-xtl6s"]
Oct 30 22:50:00.777197 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:00.777158    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8" path="/var/lib/kubelet/pods/23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8/volumes"
Oct 30 22:50:00.846010 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:00.845973    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:02.543154 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:02.543114    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:03.326952 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:03.326911    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"
Oct 30 22:50:03.340404 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:03.340367    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"
Oct 30 22:50:03.634392 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:03.634298    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"
Oct 30 22:50:04.027980 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:04.027942    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:04.591376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:04.591332    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:05.042156 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:05.042107    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:05.227993 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:05.227955    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:06.504331 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:06.504299    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:07.877021 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:07.876988    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv"
Oct 30 22:50:07.987849 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:07.987813    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ceb7603-9c15-4c92-9dbd-bd6c45828ef8-kube-api-access-kd8cj\" (UniqueName: \"kubernetes.io/projected/8ceb7603-9c15-4c92-9dbd-bd6c45828ef8-kube-api-access-kd8cj\") pod \"8ceb7603-9c15-4c92-9dbd-bd6c45828ef8\" (UID: \"8ceb7603-9c15-4c92-9dbd-bd6c45828ef8\") "
Oct 30 22:50:07.988063 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:07.987945    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ceb7603-9c15-4c92-9dbd-bd6c45828ef8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ceb7603-9c15-4c92-9dbd-bd6c45828ef8-plugins-config-volume\") pod \"8ceb7603-9c15-4c92-9dbd-bd6c45828ef8\" (UID: \"8ceb7603-9c15-4c92-9dbd-bd6c45828ef8\") "
Oct 30 22:50:07.988360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:07.988333    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ceb7603-9c15-4c92-9dbd-bd6c45828ef8-plugins-config-volume" pod "8ceb7603-9c15-4c92-9dbd-bd6c45828ef8" (UID: "8ceb7603-9c15-4c92-9dbd-bd6c45828ef8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:07.988671 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:07.988630    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ceb7603-9c15-4c92-9dbd-bd6c45828ef8-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:50:07.990464 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:07.990441    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ceb7603-9c15-4c92-9dbd-bd6c45828ef8-kube-api-access-kd8cj" pod "8ceb7603-9c15-4c92-9dbd-bd6c45828ef8" (UID: "8ceb7603-9c15-4c92-9dbd-bd6c45828ef8"). InnerVolumeSpecName "kube-api-access-kd8cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:08.089720 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:08.089685    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd8cj\" (UniqueName: \"kubernetes.io/projected/8ceb7603-9c15-4c92-9dbd-bd6c45828ef8-kube-api-access-kd8cj\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:50:08.223790 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:08.223700    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:08.671181 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:08.671147    4889 generic.go:356] "Generic (PLEG): container finished" podID="8ceb7603-9c15-4c92-9dbd-bd6c45828ef8" containerID="20a81b12ccb2708cb2029fbd1674c2150741ed20d5a46bbbc0b72705a9a9efd0" exitCode=0
Oct 30 22:50:08.671405 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:08.671195    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv"
Oct 30 22:50:08.671405 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:08.671245    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv" event={"ID":"8ceb7603-9c15-4c92-9dbd-bd6c45828ef8","Type":"ContainerDied","Data":"20a81b12ccb2708cb2029fbd1674c2150741ed20d5a46bbbc0b72705a9a9efd0"}
Oct 30 22:50:08.671405 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:08.671308    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv" event={"ID":"8ceb7603-9c15-4c92-9dbd-bd6c45828ef8","Type":"ContainerDied","Data":"6773f63d7c3a13aac86f464fa2354be67baa7d4a3d0034244f7e7c9b87ab0d38"}
Oct 30 22:50:08.671405 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:08.671331    4889 scope.go:122] "RemoveContainer" containerID="20a81b12ccb2708cb2029fbd1674c2150741ed20d5a46bbbc0b72705a9a9efd0"
Oct 30 22:50:08.685395 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:08.685373    4889 scope.go:122] "RemoveContainer" containerID="20a81b12ccb2708cb2029fbd1674c2150741ed20d5a46bbbc0b72705a9a9efd0"
Oct 30 22:50:08.685752 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:50:08.685731    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20a81b12ccb2708cb2029fbd1674c2150741ed20d5a46bbbc0b72705a9a9efd0\": container with ID starting with 20a81b12ccb2708cb2029fbd1674c2150741ed20d5a46bbbc0b72705a9a9efd0 not found: ID does not exist" containerID="20a81b12ccb2708cb2029fbd1674c2150741ed20d5a46bbbc0b72705a9a9efd0"
Oct 30 22:50:08.685855 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:08.685759    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20a81b12ccb2708cb2029fbd1674c2150741ed20d5a46bbbc0b72705a9a9efd0"} err="failed to get container status \"20a81b12ccb2708cb2029fbd1674c2150741ed20d5a46bbbc0b72705a9a9efd0\": rpc error: code = NotFound desc = could not find container \"20a81b12ccb2708cb2029fbd1674c2150741ed20d5a46bbbc0b72705a9a9efd0\": container with ID starting with 20a81b12ccb2708cb2029fbd1674c2150741ed20d5a46bbbc0b72705a9a9efd0 not found: ID does not exist"
Oct 30 22:50:08.728715 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:08.728672    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv"]
Oct 30 22:50:08.742770 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:08.742729    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv"]
Oct 30 22:50:08.777419 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:08.777383    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ceb7603-9c15-4c92-9dbd-bd6c45828ef8" path="/var/lib/kubelet/pods/8ceb7603-9c15-4c92-9dbd-bd6c45828ef8/volumes"
Oct 30 22:50:08.828930 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:08.828885    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-867f12-66d87c857f-vsjkv" podUID="8ceb7603-9c15-4c92-9dbd-bd6c45828ef8" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.176:50052\" within 1s: context deadline exceeded"
Oct 30 22:50:08.829106 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:50:08.828977    4889 logging.go:55] [core] [Channel #12371 SubChannel #12372]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.176:50052", ServerName: "10.132.0.176:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.176:50052: operation was canceled"
Oct 30 22:50:09.915002 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:09.914966    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:10.304494 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:50:10.304452    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:50:11.633384 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:11.633343    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:11.812597 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:11.812561    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:11.983026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:11.982912    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"]
Oct 30 22:50:11.983292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:11.983192    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv" podUID="6e842f91-5282-4f41-b149-d8ce4c8625a5" containerName="istio-proxy" containerID="cri-o://4863c9013c0a4784f630af6862219f193752101b94d196ab5c9daf94f22cb2a5" gracePeriod=30
Oct 30 22:50:12.086066 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:12.084802    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:12.812490 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:12.812446    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:13.298677 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:13.298638    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:13.498257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:13.481587    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:15.186784 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.186734    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:15.501302 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.501257    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"]
Oct 30 22:50:15.502464 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.502440    4889 cpu_manager.go:414] "containerMap: removing container" podUID="23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8" containerName="istio-proxy"
Oct 30 22:50:15.502524 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.502468    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8" containerName="istio-proxy"
Oct 30 22:50:15.502561 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.502520    4889 cpu_manager.go:414] "containerMap: removing container" podUID="8ceb7603-9c15-4c92-9dbd-bd6c45828ef8" containerName="payload-processing"
Oct 30 22:50:15.502561 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.502530    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ceb7603-9c15-4c92-9dbd-bd6c45828ef8" containerName="payload-processing"
Oct 30 22:50:15.502762 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.502750    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ceb7603-9c15-4c92-9dbd-bd6c45828ef8" containerName="payload-processing"
Oct 30 22:50:15.502798 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.502781    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="23ab0e7b-53c9-4cc3-ba1c-8e3f8f3ac5a8" containerName="istio-proxy"
Oct 30 22:50:15.511472 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.511139    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"
Oct 30 22:50:15.525298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.525256    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"]
Oct 30 22:50:15.531932 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.531901    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-6af86e\"/\"default-dockercfg-gjrmr\""
Oct 30 22:50:15.545947 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.545908    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-6af86e\"/\"auth-test-model-6af86e-kserve-self-signed-certs\""
Oct 30 22:50:15.545947 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.545913    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-6af86e\"/\"openshift-service-ca.crt\""
Oct 30 22:50:15.546180 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.545944    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-6af86e\"/\"kube-root-ca.crt\""
Oct 30 22:50:15.609949 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.609905    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-dshm\") pod \"auth-test-model-6af86e-kserve-568b8dc989-cdkfx\" (UID: \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\") " pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"
Oct 30 22:50:15.610203 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.610048    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-model-cache\") pod \"auth-test-model-6af86e-kserve-568b8dc989-cdkfx\" (UID: \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\") " pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"
Oct 30 22:50:15.610342 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.610283    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0a827e1-5ee3-49bf-af24-c538d0c211ef-tls-certs\") pod \"auth-test-model-6af86e-kserve-568b8dc989-cdkfx\" (UID: \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\") " pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"
Oct 30 22:50:15.610342 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.610328    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-home\") pod \"auth-test-model-6af86e-kserve-568b8dc989-cdkfx\" (UID: \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\") " pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"
Oct 30 22:50:15.610460 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.610395    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-tmp-dir\") pod \"auth-test-model-6af86e-kserve-568b8dc989-cdkfx\" (UID: \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\") " pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"
Oct 30 22:50:15.610460 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.610416    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlvbh\" (UniqueName: \"kubernetes.io/projected/f0a827e1-5ee3-49bf-af24-c538d0c211ef-kube-api-access-vlvbh\") pod \"auth-test-model-6af86e-kserve-568b8dc989-cdkfx\" (UID: \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\") " pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"
Oct 30 22:50:15.617073 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.617041    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:15.711942 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.711893    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0a827e1-5ee3-49bf-af24-c538d0c211ef-tls-certs\") pod \"auth-test-model-6af86e-kserve-568b8dc989-cdkfx\" (UID: \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\") " pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"
Oct 30 22:50:15.712170 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.711954    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-home\") pod \"auth-test-model-6af86e-kserve-568b8dc989-cdkfx\" (UID: \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\") " pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"
Oct 30 22:50:15.712170 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.711982    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-tmp-dir\") pod \"auth-test-model-6af86e-kserve-568b8dc989-cdkfx\" (UID: \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\") " pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"
Oct 30 22:50:15.712170 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.712004    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlvbh\" (UniqueName: \"kubernetes.io/projected/f0a827e1-5ee3-49bf-af24-c538d0c211ef-kube-api-access-vlvbh\") pod \"auth-test-model-6af86e-kserve-568b8dc989-cdkfx\" (UID: \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\") " pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"
Oct 30 22:50:15.712170 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.712040    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-dshm\") pod \"auth-test-model-6af86e-kserve-568b8dc989-cdkfx\" (UID: \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\") " pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"
Oct 30 22:50:15.712170 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.712087    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-model-cache\") pod \"auth-test-model-6af86e-kserve-568b8dc989-cdkfx\" (UID: \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\") " pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"
Oct 30 22:50:15.712520 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.712497    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-tmp-dir\") pod \"auth-test-model-6af86e-kserve-568b8dc989-cdkfx\" (UID: \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\") " pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"
Oct 30 22:50:15.712574 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.712506    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-home\") pod \"auth-test-model-6af86e-kserve-568b8dc989-cdkfx\" (UID: \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\") " pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"
Oct 30 22:50:15.712622 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.712601    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-model-cache\") pod \"auth-test-model-6af86e-kserve-568b8dc989-cdkfx\" (UID: \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\") " pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"
Oct 30 22:50:15.714644 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.714621    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-dshm\") pod \"auth-test-model-6af86e-kserve-568b8dc989-cdkfx\" (UID: \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\") " pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"
Oct 30 22:50:15.714876 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.714855    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0a827e1-5ee3-49bf-af24-c538d0c211ef-tls-certs\") pod \"auth-test-model-6af86e-kserve-568b8dc989-cdkfx\" (UID: \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\") " pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"
Oct 30 22:50:15.733802 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.733754    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlvbh\" (UniqueName: \"kubernetes.io/projected/f0a827e1-5ee3-49bf-af24-c538d0c211ef-kube-api-access-vlvbh\") pod \"auth-test-model-6af86e-kserve-568b8dc989-cdkfx\" (UID: \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\") " pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"
Oct 30 22:50:15.845566 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:15.845463    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"
Oct 30 22:50:16.160553 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:50:16.160520    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0a827e1_5ee3_49bf_af24_c538d0c211ef.slice/crio-3c80346bdb648e28e03a7d2a94101f425daa29eecbd0a071b44ade53d865dc20 WatchSource:0}: Error finding container 3c80346bdb648e28e03a7d2a94101f425daa29eecbd0a071b44ade53d865dc20: Status 404 returned error can't find the container with id 3c80346bdb648e28e03a7d2a94101f425daa29eecbd0a071b44ade53d865dc20
Oct 30 22:50:16.161338 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:16.161312    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"]
Oct 30 22:50:16.738473 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:16.738436    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx" event={"ID":"f0a827e1-5ee3-49bf-af24-c538d0c211ef","Type":"ContainerStarted","Data":"a04684de8e2a6aecde14261eecdc5ab9f895de0e85437fa93a173029bfb8aab9"}
Oct 30 22:50:16.738473 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:16.738478    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx" event={"ID":"f0a827e1-5ee3-49bf-af24-c538d0c211ef","Type":"ContainerStarted","Data":"3c80346bdb648e28e03a7d2a94101f425daa29eecbd0a071b44ade53d865dc20"}
Oct 30 22:50:16.782876 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:16.782743    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx" podStartSLOduration=1.782729155 podStartE2EDuration="1.782729155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:16.780275827 +0000 UTC m=+2602.136642141" watchObservedRunningTime="2026-10-30 22:50:16.782729155 +0000 UTC m=+2602.139095468"
Oct 30 22:50:16.846475 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:16.846428    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"
Oct 30 22:50:16.943859 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:16.943799    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:17.278473 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.278445    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:50:17.438802 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.438768    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e842f91-5282-4f41-b149-d8ce4c8625a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e842f91-5282-4f41-b149-d8ce4c8625a5-istiod-ca-cert\") pod \"6e842f91-5282-4f41-b149-d8ce4c8625a5\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") "
Oct 30 22:50:17.439010 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.438821    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e842f91-5282-4f41-b149-d8ce4c8625a5-kube-api-access-l67dj\" (UniqueName: \"kubernetes.io/projected/6e842f91-5282-4f41-b149-d8ce4c8625a5-kube-api-access-l67dj\") pod \"6e842f91-5282-4f41-b149-d8ce4c8625a5\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") "
Oct 30 22:50:17.439010 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.438847    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-credential-socket\") pod \"6e842f91-5282-4f41-b149-d8ce4c8625a5\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") "
Oct 30 22:50:17.439010 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.438878    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-podinfo\") pod \"6e842f91-5282-4f41-b149-d8ce4c8625a5\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") "
Oct 30 22:50:17.439010 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.438925    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-envoy\") pod \"6e842f91-5282-4f41-b149-d8ce4c8625a5\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") "
Oct 30 22:50:17.439010 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.438982    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-data\") pod \"6e842f91-5282-4f41-b149-d8ce4c8625a5\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") "
Oct 30 22:50:17.439337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.439126    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-workload-certs\") pod \"6e842f91-5282-4f41-b149-d8ce4c8625a5\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") "
Oct 30 22:50:17.439337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.439185    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-workload-socket\") pod \"6e842f91-5282-4f41-b149-d8ce4c8625a5\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") "
Oct 30 22:50:17.439337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.439273    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-token\" (UniqueName: \"kubernetes.io/projected/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-token\") pod \"6e842f91-5282-4f41-b149-d8ce4c8625a5\" (UID: \"6e842f91-5282-4f41-b149-d8ce4c8625a5\") "
Oct 30 22:50:17.439337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.439314    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-credential-socket" pod "6e842f91-5282-4f41-b149-d8ce4c8625a5" (UID: "6e842f91-5282-4f41-b149-d8ce4c8625a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:17.439571 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.439389    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-workload-certs" pod "6e842f91-5282-4f41-b149-d8ce4c8625a5" (UID: "6e842f91-5282-4f41-b149-d8ce4c8625a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:17.439571 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.439433    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e842f91-5282-4f41-b149-d8ce4c8625a5-istiod-ca-cert" pod "6e842f91-5282-4f41-b149-d8ce4c8625a5" (UID: "6e842f91-5282-4f41-b149-d8ce4c8625a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:17.440267 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.440206    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e842f91-5282-4f41-b149-d8ce4c8625a5-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:50:17.440409 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.440270    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:50:17.440409 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.440285    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:50:17.440409 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.440348    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-data" pod "6e842f91-5282-4f41-b149-d8ce4c8625a5" (UID: "6e842f91-5282-4f41-b149-d8ce4c8625a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:17.440621 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.440409    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-workload-socket" pod "6e842f91-5282-4f41-b149-d8ce4c8625a5" (UID: "6e842f91-5282-4f41-b149-d8ce4c8625a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:17.441894 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.441871    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e842f91-5282-4f41-b149-d8ce4c8625a5-kube-api-access-l67dj" pod "6e842f91-5282-4f41-b149-d8ce4c8625a5" (UID: "6e842f91-5282-4f41-b149-d8ce4c8625a5"). InnerVolumeSpecName "kube-api-access-l67dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:17.441994 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.441889    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-envoy" pod "6e842f91-5282-4f41-b149-d8ce4c8625a5" (UID: "6e842f91-5282-4f41-b149-d8ce4c8625a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:17.441994 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.441906    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-token" pod "6e842f91-5282-4f41-b149-d8ce4c8625a5" (UID: "6e842f91-5282-4f41-b149-d8ce4c8625a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:17.442405 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.442385    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-podinfo" pod "6e842f91-5282-4f41-b149-d8ce4c8625a5" (UID: "6e842f91-5282-4f41-b149-d8ce4c8625a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:50:17.527782 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.527666    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:17.541243 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.541194    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:50:17.541243 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.541242    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:50:17.541471 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.541254    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l67dj\" (UniqueName: \"kubernetes.io/projected/6e842f91-5282-4f41-b149-d8ce4c8625a5-kube-api-access-l67dj\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:50:17.541471 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.541265    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:50:17.541471 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.541274    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:50:17.541471 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.541282    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e842f91-5282-4f41-b149-d8ce4c8625a5-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:50:17.696205 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.696168    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:17.749776 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.749728    4889 generic.go:356] "Generic (PLEG): container finished" podID="6e842f91-5282-4f41-b149-d8ce4c8625a5" containerID="4863c9013c0a4784f630af6862219f193752101b94d196ab5c9daf94f22cb2a5" exitCode=0
Oct 30 22:50:17.750282 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.749790    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"
Oct 30 22:50:17.750282 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.749816    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv" event={"ID":"6e842f91-5282-4f41-b149-d8ce4c8625a5","Type":"ContainerDied","Data":"4863c9013c0a4784f630af6862219f193752101b94d196ab5c9daf94f22cb2a5"}
Oct 30 22:50:17.750282 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.749885    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv" event={"ID":"6e842f91-5282-4f41-b149-d8ce4c8625a5","Type":"ContainerDied","Data":"c925739945a5ddf8fce4d02edaacf0c64dfaeb18f7fa5df950d1ab6961460b0c"}
Oct 30 22:50:17.750282 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.749915    4889 scope.go:122] "RemoveContainer" containerID="4863c9013c0a4784f630af6862219f193752101b94d196ab5c9daf94f22cb2a5"
Oct 30 22:50:17.764810 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.764781    4889 scope.go:122] "RemoveContainer" containerID="4863c9013c0a4784f630af6862219f193752101b94d196ab5c9daf94f22cb2a5"
Oct 30 22:50:17.765171 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:50:17.765144    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4863c9013c0a4784f630af6862219f193752101b94d196ab5c9daf94f22cb2a5\": container with ID starting with 4863c9013c0a4784f630af6862219f193752101b94d196ab5c9daf94f22cb2a5 not found: ID does not exist" containerID="4863c9013c0a4784f630af6862219f193752101b94d196ab5c9daf94f22cb2a5"
Oct 30 22:50:17.765244 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.765183    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4863c9013c0a4784f630af6862219f193752101b94d196ab5c9daf94f22cb2a5"} err="failed to get container status \"4863c9013c0a4784f630af6862219f193752101b94d196ab5c9daf94f22cb2a5\": rpc error: code = NotFound desc = could not find container \"4863c9013c0a4784f630af6862219f193752101b94d196ab5c9daf94f22cb2a5\": container with ID starting with 4863c9013c0a4784f630af6862219f193752101b94d196ab5c9daf94f22cb2a5 not found: ID does not exist"
Oct 30 22:50:17.807036 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.806951    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"]
Oct 30 22:50:17.813074 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:17.813042    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-9rbbv"]
Oct 30 22:50:18.776680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:18.776608    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e842f91-5282-4f41-b149-d8ce4c8625a5" path="/var/lib/kubelet/pods/6e842f91-5282-4f41-b149-d8ce4c8625a5/volumes"
Oct 30 22:50:19.059347 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:19.059252    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:19.259421 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:19.259382    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:19.441161 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:19.440974    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:19.505001 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:19.504831    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5"]
Oct 30 22:50:19.506587 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:19.506554    4889 cpu_manager.go:414] "containerMap: removing container" podUID="6e842f91-5282-4f41-b149-d8ce4c8625a5" containerName="istio-proxy"
Oct 30 22:50:19.506587 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:19.506586    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e842f91-5282-4f41-b149-d8ce4c8625a5" containerName="istio-proxy"
Oct 30 22:50:19.506787 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:19.506775    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e842f91-5282-4f41-b149-d8ce4c8625a5" containerName="istio-proxy"
Oct 30 22:50:19.514451 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:19.514424    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5"
Oct 30 22:50:19.515252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:19.515208    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5"]
Oct 30 22:50:19.531538 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:19.531498    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-cdc122\""
Oct 30 22:50:19.548411 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:19.548379    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-cdc122-dockercfg-prrvz\""
Oct 30 22:50:19.573583 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:19.573537    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dcnf\" (UniqueName: \"kubernetes.io/projected/e1595760-0ede-4e19-ad58-741f6f577207-kube-api-access-6dcnf\") pod \"payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5\" (UID: \"e1595760-0ede-4e19-ad58-741f6f577207\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5"
Oct 30 22:50:19.573811 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:19.573686    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1595760-0ede-4e19-ad58-741f6f577207-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5\" (UID: \"e1595760-0ede-4e19-ad58-741f6f577207\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5"
Oct 30 22:50:19.675217 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:19.675178    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dcnf\" (UniqueName: \"kubernetes.io/projected/e1595760-0ede-4e19-ad58-741f6f577207-kube-api-access-6dcnf\") pod \"payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5\" (UID: \"e1595760-0ede-4e19-ad58-741f6f577207\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5"
Oct 30 22:50:19.675452 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:19.675311    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1595760-0ede-4e19-ad58-741f6f577207-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5\" (UID: \"e1595760-0ede-4e19-ad58-741f6f577207\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5"
Oct 30 22:50:19.676064 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:19.676030    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1595760-0ede-4e19-ad58-741f6f577207-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5\" (UID: \"e1595760-0ede-4e19-ad58-741f6f577207\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5"
Oct 30 22:50:19.697396 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:19.697322    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dcnf\" (UniqueName: \"kubernetes.io/projected/e1595760-0ede-4e19-ad58-741f6f577207-kube-api-access-6dcnf\") pod \"payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5\" (UID: \"e1595760-0ede-4e19-ad58-741f6f577207\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5"
Oct 30 22:50:19.835199 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:19.835166    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5"
Oct 30 22:50:20.142531 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:20.142500    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5"]
Oct 30 22:50:20.142531 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:50:20.142520    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1595760_0ede_4e19_ad58_741f6f577207.slice/crio-830d1d4db961e5de1e57f549e10082de79f65b3848dbdab774aa4e0cfcab63c3 WatchSource:0}: Error finding container 830d1d4db961e5de1e57f549e10082de79f65b3848dbdab774aa4e0cfcab63c3: Status 404 returned error can't find the container with id 830d1d4db961e5de1e57f549e10082de79f65b3848dbdab774aa4e0cfcab63c3
Oct 30 22:50:20.721515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:20.721475    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:20.776914 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:20.776872    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5" event={"ID":"e1595760-0ede-4e19-ad58-741f6f577207","Type":"ContainerStarted","Data":"290e72c21f8ca404ec8317466ff1623eed8e702e7f36a523208ad6bd9e2d6398"}
Oct 30 22:50:20.777068 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:20.776932    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5" event={"ID":"e1595760-0ede-4e19-ad58-741f6f577207","Type":"ContainerStarted","Data":"830d1d4db961e5de1e57f549e10082de79f65b3848dbdab774aa4e0cfcab63c3"}
Oct 30 22:50:20.777068 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:20.777009    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5"
Oct 30 22:50:20.840899 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:20.840834    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5" podStartSLOduration=1.840818501 podStartE2EDuration="1.840818501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:20.836689281 +0000 UTC m=+2606.193055599" watchObservedRunningTime="2026-10-30 22:50:20.840818501 +0000 UTC m=+2606.197184813"
Oct 30 22:50:22.440668 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:22.440625    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:24.350306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:24.350258    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:24.570178 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:24.570127    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:25.846630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:25.846585    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"
Oct 30 22:50:25.860317 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:25.860285    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"
Oct 30 22:50:26.222802 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:26.222633    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:26.840777 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:26.840745    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"
Oct 30 22:50:27.202044 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:27.201944    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:28.079182 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:28.079147    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:29.802671 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:29.802638    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:31.426536 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:31.426500    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:31.796470 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:31.796434    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5"
Oct 30 22:50:33.143009 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:33.142973    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:34.818622 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:34.818587    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:36.465293 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:36.465255    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:38.104292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:38.104251    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:39.803589 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:39.803550    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:40.305148 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:50:40.305106    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:50:41.537278 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:41.537209    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:43.316446 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.316384    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:43.580727 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.580638    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"]
Oct 30 22:50:43.587019 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.586988    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"
Oct 30 22:50:43.613589 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.613539    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"]
Oct 30 22:50:43.615576 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.615534    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca61e410-bf75-487e-b37a-333a1141ffb6-tls-certs\") pod \"sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b\" (UID: \"ca61e410-bf75-487e-b37a-333a1141ffb6\") " pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"
Oct 30 22:50:43.615767 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.615606    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hpv4\" (UniqueName: \"kubernetes.io/projected/ca61e410-bf75-487e-b37a-333a1141ffb6-kube-api-access-5hpv4\") pod \"sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b\" (UID: \"ca61e410-bf75-487e-b37a-333a1141ffb6\") " pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"
Oct 30 22:50:43.615854 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.615765    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-dshm\") pod \"sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b\" (UID: \"ca61e410-bf75-487e-b37a-333a1141ffb6\") " pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"
Oct 30 22:50:43.615952 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.615857    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-tmp-dir\") pod \"sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b\" (UID: \"ca61e410-bf75-487e-b37a-333a1141ffb6\") " pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"
Oct 30 22:50:43.615952 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.615910    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-model-cache\") pod \"sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b\" (UID: \"ca61e410-bf75-487e-b37a-333a1141ffb6\") " pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"
Oct 30 22:50:43.616135 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.615970    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-home\") pod \"sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b\" (UID: \"ca61e410-bf75-487e-b37a-333a1141ffb6\") " pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"
Oct 30 22:50:43.620589 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.620545    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-3cd77f\"/\"kube-root-ca.crt\""
Oct 30 22:50:43.620810 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.620644    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-3cd77f\"/\"openshift-service-ca.crt\""
Oct 30 22:50:43.621158 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.621136    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-3cd77f\"/\"sub-test-model-3cd77f-kserve-self-signed-certs\""
Oct 30 22:50:43.621301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.621174    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-3cd77f\"/\"default-dockercfg-q6s8c\""
Oct 30 22:50:43.717204 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.717167    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-tmp-dir\") pod \"sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b\" (UID: \"ca61e410-bf75-487e-b37a-333a1141ffb6\") " pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"
Oct 30 22:50:43.717408 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.717266    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-model-cache\") pod \"sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b\" (UID: \"ca61e410-bf75-487e-b37a-333a1141ffb6\") " pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"
Oct 30 22:50:43.717408 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.717336    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-home\") pod \"sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b\" (UID: \"ca61e410-bf75-487e-b37a-333a1141ffb6\") " pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"
Oct 30 22:50:43.717408 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.717399    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca61e410-bf75-487e-b37a-333a1141ffb6-tls-certs\") pod \"sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b\" (UID: \"ca61e410-bf75-487e-b37a-333a1141ffb6\") " pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"
Oct 30 22:50:43.717534 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.717451    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hpv4\" (UniqueName: \"kubernetes.io/projected/ca61e410-bf75-487e-b37a-333a1141ffb6-kube-api-access-5hpv4\") pod \"sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b\" (UID: \"ca61e410-bf75-487e-b37a-333a1141ffb6\") " pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"
Oct 30 22:50:43.717653 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.717629    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-dshm\") pod \"sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b\" (UID: \"ca61e410-bf75-487e-b37a-333a1141ffb6\") " pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"
Oct 30 22:50:43.717801 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.717732    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-model-cache\") pod \"sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b\" (UID: \"ca61e410-bf75-487e-b37a-333a1141ffb6\") " pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"
Oct 30 22:50:43.717801 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.717703    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-tmp-dir\") pod \"sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b\" (UID: \"ca61e410-bf75-487e-b37a-333a1141ffb6\") " pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"
Oct 30 22:50:43.717900 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.717798    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-home\") pod \"sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b\" (UID: \"ca61e410-bf75-487e-b37a-333a1141ffb6\") " pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"
Oct 30 22:50:43.720137 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.720110    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-dshm\") pod \"sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b\" (UID: \"ca61e410-bf75-487e-b37a-333a1141ffb6\") " pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"
Oct 30 22:50:43.720351 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.720334    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca61e410-bf75-487e-b37a-333a1141ffb6-tls-certs\") pod \"sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b\" (UID: \"ca61e410-bf75-487e-b37a-333a1141ffb6\") " pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"
Oct 30 22:50:43.737426 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.737392    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:43.747974 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.747942    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hpv4\" (UniqueName: \"kubernetes.io/projected/ca61e410-bf75-487e-b37a-333a1141ffb6-kube-api-access-5hpv4\") pod \"sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b\" (UID: \"ca61e410-bf75-487e-b37a-333a1141ffb6\") " pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"
Oct 30 22:50:43.904493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:43.904405    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"
Oct 30 22:50:44.217058 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:50:44.217002    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca61e410_bf75_487e_b37a_333a1141ffb6.slice/crio-aefb3613f7562f8d9dbc56d4ee69368b056e066c645f23b125b0ab0649f89fb2 WatchSource:0}: Error finding container aefb3613f7562f8d9dbc56d4ee69368b056e066c645f23b125b0ab0649f89fb2: Status 404 returned error can't find the container with id aefb3613f7562f8d9dbc56d4ee69368b056e066c645f23b125b0ab0649f89fb2
Oct 30 22:50:44.217399 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:44.217376    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"]
Oct 30 22:50:44.978493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:44.978456    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b" event={"ID":"ca61e410-bf75-487e-b37a-333a1141ffb6","Type":"ContainerStarted","Data":"104c0e2d926ff6d194443d761396827155528998e02ee40ccc137a016a324de7"}
Oct 30 22:50:44.979053 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:44.978506    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b" event={"ID":"ca61e410-bf75-487e-b37a-333a1141ffb6","Type":"ContainerStarted","Data":"aefb3613f7562f8d9dbc56d4ee69368b056e066c645f23b125b0ab0649f89fb2"}
Oct 30 22:50:45.030324 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:45.030257    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b" podStartSLOduration=2.030218057 podStartE2EDuration="2.030218057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:45.026490462 +0000 UTC m=+2630.382856778" watchObservedRunningTime="2026-10-30 22:50:45.030218057 +0000 UTC m=+2630.386584371"
Oct 30 22:50:45.113835 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:45.113801    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:45.904858 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:45.904820    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"
Oct 30 22:50:46.991339 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:46.991295    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:48.722957 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:48.722898    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:50.134672 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:50.134629    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:50.308841 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:50.308591    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:50.479123 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:50.479032    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:51.059134 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:51.059095    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"]
Oct 30 22:50:51.153573 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:51.153523    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:52.274549 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:52.274508    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:52.712859 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:52.712765    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:53.904792 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:53.904757    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"
Oct 30 22:50:53.919241 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:53.919186    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"
Oct 30 22:50:54.069342 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:54.069299    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"
Oct 30 22:50:54.189071 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:54.188961    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:56.144139 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:56.144101    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:56.890141 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:56.890101    4889 scope.go:122] "RemoveContainer" containerID="e8c2089231571ba84c3f9802a2814d7d3c1902509eea0e82f6b9d3e07722e922"
Oct 30 22:50:57.857539 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:57.857501    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:59.530574 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:50:59.530527    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:01.231304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:01.231271    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:03.048269 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:03.048192    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:04.932540 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:04.932452    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:06.081834 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:06.081793    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj" podUID="21e5ac2e-104f-4c24-8e80-6fd0b457478d" containerName="main" containerID="cri-o://3931135a1b5b9a9a39f7fc2e1515075f56049e371b0f2553291202242034c169" gracePeriod=45
Oct 30 22:51:06.383932 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:06.383902    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"
Oct 30 22:51:06.477137 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:06.477106    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21e5ac2e-104f-4c24-8e80-6fd0b457478d-tls-certs\" (UniqueName: \"kubernetes.io/secret/21e5ac2e-104f-4c24-8e80-6fd0b457478d-tls-certs\") pod \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\" (UID: \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\") "
Oct 30 22:51:06.477386 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:06.477166    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21e5ac2e-104f-4c24-8e80-6fd0b457478d-kube-api-access-857f2\" (UniqueName: \"kubernetes.io/projected/21e5ac2e-104f-4c24-8e80-6fd0b457478d-kube-api-access-857f2\") pod \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\" (UID: \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\") "
Oct 30 22:51:06.477386 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:06.477338    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-home\" (UniqueName: \"kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-home\") pod \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\" (UID: \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\") "
Oct 30 22:51:06.477526 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:06.477509    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-model-cache\") pod \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\" (UID: \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\") "
Oct 30 22:51:06.477582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:06.477550    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-tmp-dir\") pod \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\" (UID: \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\") "
Oct 30 22:51:06.477676 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:06.477622    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-dshm\") pod \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\" (UID: \"21e5ac2e-104f-4c24-8e80-6fd0b457478d\") "
Oct 30 22:51:06.477742 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:06.477643    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-home" pod "21e5ac2e-104f-4c24-8e80-6fd0b457478d" (UID: "21e5ac2e-104f-4c24-8e80-6fd0b457478d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:06.477861 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:06.477838    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-model-cache" pod "21e5ac2e-104f-4c24-8e80-6fd0b457478d" (UID: "21e5ac2e-104f-4c24-8e80-6fd0b457478d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:06.477978 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:06.477924    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-tmp-dir" pod "21e5ac2e-104f-4c24-8e80-6fd0b457478d" (UID: "21e5ac2e-104f-4c24-8e80-6fd0b457478d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:06.478649 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:06.478624    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:51:06.478727 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:06.478655    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:51:06.478727 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:06.478670    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:51:06.479947 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:06.479919    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21e5ac2e-104f-4c24-8e80-6fd0b457478d-tls-certs" pod "21e5ac2e-104f-4c24-8e80-6fd0b457478d" (UID: "21e5ac2e-104f-4c24-8e80-6fd0b457478d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:51:06.480046 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:06.480013    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-dshm" pod "21e5ac2e-104f-4c24-8e80-6fd0b457478d" (UID: "21e5ac2e-104f-4c24-8e80-6fd0b457478d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:06.480133 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:06.480116    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e5ac2e-104f-4c24-8e80-6fd0b457478d-kube-api-access-857f2" pod "21e5ac2e-104f-4c24-8e80-6fd0b457478d" (UID: "21e5ac2e-104f-4c24-8e80-6fd0b457478d"). InnerVolumeSpecName "kube-api-access-857f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:06.580661 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:06.580619    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21e5ac2e-104f-4c24-8e80-6fd0b457478d-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:51:06.580661 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:06.580656    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21e5ac2e-104f-4c24-8e80-6fd0b457478d-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:51:06.580661 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:06.580669    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-857f2\" (UniqueName: \"kubernetes.io/projected/21e5ac2e-104f-4c24-8e80-6fd0b457478d-kube-api-access-857f2\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:51:06.623536 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:06.623437    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:06.800698 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:06.800655    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:07.159418 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:07.159371    4889 generic.go:356] "Generic (PLEG): container finished" podID="21e5ac2e-104f-4c24-8e80-6fd0b457478d" containerID="3931135a1b5b9a9a39f7fc2e1515075f56049e371b0f2553291202242034c169" exitCode=0
Oct 30 22:51:07.159923 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:07.159450    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"
Oct 30 22:51:07.159923 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:07.159460    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj" event={"ID":"21e5ac2e-104f-4c24-8e80-6fd0b457478d","Type":"ContainerDied","Data":"3931135a1b5b9a9a39f7fc2e1515075f56049e371b0f2553291202242034c169"}
Oct 30 22:51:07.159923 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:07.159528    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj" event={"ID":"21e5ac2e-104f-4c24-8e80-6fd0b457478d","Type":"ContainerDied","Data":"5f52f602a008eccae8a1410c2b05299193136511346fa863de36d33c7af09efb"}
Oct 30 22:51:07.159923 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:07.159551    4889 scope.go:122] "RemoveContainer" containerID="3931135a1b5b9a9a39f7fc2e1515075f56049e371b0f2553291202242034c169"
Oct 30 22:51:07.171920 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:07.171895    4889 scope.go:122] "RemoveContainer" containerID="3931135a1b5b9a9a39f7fc2e1515075f56049e371b0f2553291202242034c169"
Oct 30 22:51:07.172295 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:51:07.172271    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3931135a1b5b9a9a39f7fc2e1515075f56049e371b0f2553291202242034c169\": container with ID starting with 3931135a1b5b9a9a39f7fc2e1515075f56049e371b0f2553291202242034c169 not found: ID does not exist" containerID="3931135a1b5b9a9a39f7fc2e1515075f56049e371b0f2553291202242034c169"
Oct 30 22:51:07.172397 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:07.172304    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3931135a1b5b9a9a39f7fc2e1515075f56049e371b0f2553291202242034c169"} err="failed to get container status \"3931135a1b5b9a9a39f7fc2e1515075f56049e371b0f2553291202242034c169\": rpc error: code = NotFound desc = could not find container \"3931135a1b5b9a9a39f7fc2e1515075f56049e371b0f2553291202242034c169\": container with ID starting with 3931135a1b5b9a9a39f7fc2e1515075f56049e371b0f2553291202242034c169 not found: ID does not exist"
Oct 30 22:51:07.217155 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:07.217117    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"]
Oct 30 22:51:07.230678 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:07.230641    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-0ff5b0/auth-test-model-0ff5b0-kserve-7df6875549-2nbwj"]
Oct 30 22:51:08.344672 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:08.344633    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:08.777689 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:08.777652    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21e5ac2e-104f-4c24-8e80-6fd0b457478d" path="/var/lib/kubelet/pods/21e5ac2e-104f-4c24-8e80-6fd0b457478d/volumes"
Oct 30 22:51:10.241454 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:10.241413    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:10.316826 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:51:10.316782    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:51:12.052978 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:12.052944    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:13.866357 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:13.866318    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:15.611677 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:15.611638    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:17.566629 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:17.566592    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:17.661392 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:17.661353    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"]
Oct 30 22:51:17.751071 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:17.751027    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:18.363895 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:18.363847    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:18.817101 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:18.816991    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:19.294839 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:19.294799    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:19.472074 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:19.472032    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:21.035445 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:21.035403    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:22.166395 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:22.166347    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:22.335124 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:22.335082    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:22.968576 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:22.968544    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:23.987301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:23.987266    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:24.007126 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:24.007092    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc"]
Oct 30 22:51:24.007864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:24.007844    4889 cpu_manager.go:414] "containerMap: removing container" podUID="21e5ac2e-104f-4c24-8e80-6fd0b457478d" containerName="main"
Oct 30 22:51:24.007864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:24.007861    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21e5ac2e-104f-4c24-8e80-6fd0b457478d" containerName="main"
Oct 30 22:51:24.008051 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:24.008020    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="21e5ac2e-104f-4c24-8e80-6fd0b457478d" containerName="main"
Oct 30 22:51:24.011799 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:24.011776    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc"
Oct 30 22:51:24.034050 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:24.034019    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc"]
Oct 30 22:51:24.042919 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:24.042882    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e3a7dd\""
Oct 30 22:51:24.042919 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:24.042890    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e3a7dd-dockercfg-z2spv\""
Oct 30 22:51:24.148317 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:24.148264    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdk6t\" (UniqueName: \"kubernetes.io/projected/879f116a-b7d8-4db8-af80-e76b9018c22f-kube-api-access-tdk6t\") pod \"payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc\" (UID: \"879f116a-b7d8-4db8-af80-e76b9018c22f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc"
Oct 30 22:51:24.148722 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:24.148689    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879f116a-b7d8-4db8-af80-e76b9018c22f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc\" (UID: \"879f116a-b7d8-4db8-af80-e76b9018c22f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc"
Oct 30 22:51:24.188554 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:24.188517    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:24.250513 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:24.250475    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdk6t\" (UniqueName: \"kubernetes.io/projected/879f116a-b7d8-4db8-af80-e76b9018c22f-kube-api-access-tdk6t\") pod \"payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc\" (UID: \"879f116a-b7d8-4db8-af80-e76b9018c22f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc"
Oct 30 22:51:24.250716 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:24.250546    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879f116a-b7d8-4db8-af80-e76b9018c22f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc\" (UID: \"879f116a-b7d8-4db8-af80-e76b9018c22f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc"
Oct 30 22:51:24.251301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:24.251274    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879f116a-b7d8-4db8-af80-e76b9018c22f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc\" (UID: \"879f116a-b7d8-4db8-af80-e76b9018c22f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc"
Oct 30 22:51:24.272357 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:24.272325    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdk6t\" (UniqueName: \"kubernetes.io/projected/879f116a-b7d8-4db8-af80-e76b9018c22f-kube-api-access-tdk6t\") pod \"payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc\" (UID: \"879f116a-b7d8-4db8-af80-e76b9018c22f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc"
Oct 30 22:51:24.328404 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:24.328352    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc"
Oct 30 22:51:24.639350 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:51:24.639318    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod879f116a_b7d8_4db8_af80_e76b9018c22f.slice/crio-e8df9f557ac30b70a99c576540c69b542a7f7c7053a0dd3ce2a592034ce000c2 WatchSource:0}: Error finding container e8df9f557ac30b70a99c576540c69b542a7f7c7053a0dd3ce2a592034ce000c2: Status 404 returned error can't find the container with id e8df9f557ac30b70a99c576540c69b542a7f7c7053a0dd3ce2a592034ce000c2
Oct 30 22:51:24.639963 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:24.639864    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc"]
Oct 30 22:51:24.873610 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:24.873572    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:25.325576 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:25.325529    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc" event={"ID":"879f116a-b7d8-4db8-af80-e76b9018c22f","Type":"ContainerStarted","Data":"48a9771e69e02858520c90113581205bff49e9dca7c3cd9cff19debc8c96330c"}
Oct 30 22:51:25.326025 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:25.325599    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc" event={"ID":"879f116a-b7d8-4db8-af80-e76b9018c22f","Type":"ContainerStarted","Data":"e8df9f557ac30b70a99c576540c69b542a7f7c7053a0dd3ce2a592034ce000c2"}
Oct 30 22:51:25.326025 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:25.325624    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc"
Oct 30 22:51:25.368797 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:25.368727    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc" podStartSLOduration=2.36870696 podStartE2EDuration="2.36870696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:25.367532178 +0000 UTC m=+2670.723898495" watchObservedRunningTime="2026-10-30 22:51:25.36870696 +0000 UTC m=+2670.725073277"
Oct 30 22:51:26.104343 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.104308    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"]
Oct 30 22:51:26.109359 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.109322    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"
Oct 30 22:51:26.142515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.142482    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-bed544\"/\"default-dockercfg-xg4wc\""
Oct 30 22:51:26.142515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.142500    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-bed544\"/\"openshift-service-ca.crt\""
Oct 30 22:51:26.142770 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.142538    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-bed544\"/\"kube-root-ca.crt\""
Oct 30 22:51:26.142770 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.142681    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-bed544\"/\"auth-test-model-bed544-kserve-self-signed-certs\""
Oct 30 22:51:26.149321 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.149285    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"]
Oct 30 22:51:26.179212 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.179179    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f8596b1-c420-46cd-85cd-6b68dffff980-tls-certs\") pod \"auth-test-model-bed544-kserve-659fbc956f-jmcdb\" (UID: \"2f8596b1-c420-46cd-85cd-6b68dffff980\") " pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"
Oct 30 22:51:26.179442 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.179299    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lphb\" (UniqueName: \"kubernetes.io/projected/2f8596b1-c420-46cd-85cd-6b68dffff980-kube-api-access-2lphb\") pod \"auth-test-model-bed544-kserve-659fbc956f-jmcdb\" (UID: \"2f8596b1-c420-46cd-85cd-6b68dffff980\") " pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"
Oct 30 22:51:26.179442 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.179383    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-tmp-dir\") pod \"auth-test-model-bed544-kserve-659fbc956f-jmcdb\" (UID: \"2f8596b1-c420-46cd-85cd-6b68dffff980\") " pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"
Oct 30 22:51:26.179524 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.179458    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-home\") pod \"auth-test-model-bed544-kserve-659fbc956f-jmcdb\" (UID: \"2f8596b1-c420-46cd-85cd-6b68dffff980\") " pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"
Oct 30 22:51:26.179524 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.179489    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-model-cache\") pod \"auth-test-model-bed544-kserve-659fbc956f-jmcdb\" (UID: \"2f8596b1-c420-46cd-85cd-6b68dffff980\") " pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"
Oct 30 22:51:26.179682 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.179660    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-dshm\") pod \"auth-test-model-bed544-kserve-659fbc956f-jmcdb\" (UID: \"2f8596b1-c420-46cd-85cd-6b68dffff980\") " pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"
Oct 30 22:51:26.215557 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.215516    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:26.281382 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.281344    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-tmp-dir\") pod \"auth-test-model-bed544-kserve-659fbc956f-jmcdb\" (UID: \"2f8596b1-c420-46cd-85cd-6b68dffff980\") " pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"
Oct 30 22:51:26.281575 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.281395    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-home\") pod \"auth-test-model-bed544-kserve-659fbc956f-jmcdb\" (UID: \"2f8596b1-c420-46cd-85cd-6b68dffff980\") " pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"
Oct 30 22:51:26.281575 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.281417    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-model-cache\") pod \"auth-test-model-bed544-kserve-659fbc956f-jmcdb\" (UID: \"2f8596b1-c420-46cd-85cd-6b68dffff980\") " pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"
Oct 30 22:51:26.281575 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.281527    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-dshm\") pod \"auth-test-model-bed544-kserve-659fbc956f-jmcdb\" (UID: \"2f8596b1-c420-46cd-85cd-6b68dffff980\") " pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"
Oct 30 22:51:26.281715 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.281587    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f8596b1-c420-46cd-85cd-6b68dffff980-tls-certs\") pod \"auth-test-model-bed544-kserve-659fbc956f-jmcdb\" (UID: \"2f8596b1-c420-46cd-85cd-6b68dffff980\") " pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"
Oct 30 22:51:26.281715 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.281632    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lphb\" (UniqueName: \"kubernetes.io/projected/2f8596b1-c420-46cd-85cd-6b68dffff980-kube-api-access-2lphb\") pod \"auth-test-model-bed544-kserve-659fbc956f-jmcdb\" (UID: \"2f8596b1-c420-46cd-85cd-6b68dffff980\") " pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"
Oct 30 22:51:26.281851 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.281828    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-tmp-dir\") pod \"auth-test-model-bed544-kserve-659fbc956f-jmcdb\" (UID: \"2f8596b1-c420-46cd-85cd-6b68dffff980\") " pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"
Oct 30 22:51:26.281902 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.281849    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-home\") pod \"auth-test-model-bed544-kserve-659fbc956f-jmcdb\" (UID: \"2f8596b1-c420-46cd-85cd-6b68dffff980\") " pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"
Oct 30 22:51:26.281972 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.281949    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-model-cache\") pod \"auth-test-model-bed544-kserve-659fbc956f-jmcdb\" (UID: \"2f8596b1-c420-46cd-85cd-6b68dffff980\") " pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"
Oct 30 22:51:26.284188 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.284156    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-dshm\") pod \"auth-test-model-bed544-kserve-659fbc956f-jmcdb\" (UID: \"2f8596b1-c420-46cd-85cd-6b68dffff980\") " pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"
Oct 30 22:51:26.284486 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.284470    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f8596b1-c420-46cd-85cd-6b68dffff980-tls-certs\") pod \"auth-test-model-bed544-kserve-659fbc956f-jmcdb\" (UID: \"2f8596b1-c420-46cd-85cd-6b68dffff980\") " pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"
Oct 30 22:51:26.303304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.303269    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lphb\" (UniqueName: \"kubernetes.io/projected/2f8596b1-c420-46cd-85cd-6b68dffff980-kube-api-access-2lphb\") pod \"auth-test-model-bed544-kserve-659fbc956f-jmcdb\" (UID: \"2f8596b1-c420-46cd-85cd-6b68dffff980\") " pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"
Oct 30 22:51:26.428111 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.428020    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"
Oct 30 22:51:26.729991 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.729900    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:26.749937 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:26.749891    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"]
Oct 30 22:51:26.750285 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:51:26.750251    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f8596b1_c420_46cd_85cd_6b68dffff980.slice/crio-6f3962d9968201991e47034324552a427a6cdda0b3870a397a249bfd155dad79 WatchSource:0}: Error finding container 6f3962d9968201991e47034324552a427a6cdda0b3870a397a249bfd155dad79: Status 404 returned error can't find the container with id 6f3962d9968201991e47034324552a427a6cdda0b3870a397a249bfd155dad79
Oct 30 22:51:27.346089 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:27.346037    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb" event={"ID":"2f8596b1-c420-46cd-85cd-6b68dffff980","Type":"ContainerStarted","Data":"ffb19f8b6a11749f5f9fde8f2b92388f71aae2b36e28d152a2cb91678991de00"}
Oct 30 22:51:27.346333 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:27.346105    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb" event={"ID":"2f8596b1-c420-46cd-85cd-6b68dffff980","Type":"ContainerStarted","Data":"6f3962d9968201991e47034324552a427a6cdda0b3870a397a249bfd155dad79"}
Oct 30 22:51:27.397398 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:27.397339    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb" podStartSLOduration=1.397323261 podStartE2EDuration="1.397323261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:27.393977973 +0000 UTC m=+2672.750344287" watchObservedRunningTime="2026-10-30 22:51:27.397323261 +0000 UTC m=+2672.753689576"
Oct 30 22:51:27.428585 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:27.428547    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"
Oct 30 22:51:28.581636 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:28.581594    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:30.209342 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:30.209303    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:31.914199 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:31.914162    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:32.692931 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:32.692891    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks" podUID="46e8ec9f-b22e-4251-9652-118c6b52d401" containerName="main" containerID="cri-o://f63015797021c1648ab18d9eb2dccd7aa32abfd7dacb318ceb380da7f1408262" gracePeriod=45
Oct 30 22:51:32.778115 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:32.778068    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks" podUID="46e8ec9f-b22e-4251-9652-118c6b52d401" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": dial tcp 10.132.0.190:8000: connect: connection refused"
Oct 30 22:51:33.019600 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.019570    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"
Oct 30 22:51:33.124045 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.124008    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"]
Oct 30 22:51:33.124809 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.124792    4889 cpu_manager.go:414] "containerMap: removing container" podUID="46e8ec9f-b22e-4251-9652-118c6b52d401" containerName="main"
Oct 30 22:51:33.124809 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.124811    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46e8ec9f-b22e-4251-9652-118c6b52d401" containerName="main"
Oct 30 22:51:33.124961 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.124952    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="46e8ec9f-b22e-4251-9652-118c6b52d401" containerName="main"
Oct 30 22:51:33.128763 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.128739    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.145950 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.145896    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-d9f1c2-openshift-default-dockercfg-rvd79\""
Oct 30 22:51:33.150360 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.150327    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"]
Oct 30 22:51:33.190280 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.190206    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46e8ec9f-b22e-4251-9652-118c6b52d401-kube-api-access-djvv9\" (UniqueName: \"kubernetes.io/projected/46e8ec9f-b22e-4251-9652-118c6b52d401-kube-api-access-djvv9\") pod \"46e8ec9f-b22e-4251-9652-118c6b52d401\" (UID: \"46e8ec9f-b22e-4251-9652-118c6b52d401\") "
Oct 30 22:51:33.190505 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.190303    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-dshm\" (UniqueName: \"kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-dshm\") pod \"46e8ec9f-b22e-4251-9652-118c6b52d401\" (UID: \"46e8ec9f-b22e-4251-9652-118c6b52d401\") "
Oct 30 22:51:33.190505 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.190338    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-model-cache\") pod \"46e8ec9f-b22e-4251-9652-118c6b52d401\" (UID: \"46e8ec9f-b22e-4251-9652-118c6b52d401\") "
Oct 30 22:51:33.190505 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.190361    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-tmp-dir\") pod \"46e8ec9f-b22e-4251-9652-118c6b52d401\" (UID: \"46e8ec9f-b22e-4251-9652-118c6b52d401\") "
Oct 30 22:51:33.190505 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.190445    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-home\" (UniqueName: \"kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-home\") pod \"46e8ec9f-b22e-4251-9652-118c6b52d401\" (UID: \"46e8ec9f-b22e-4251-9652-118c6b52d401\") "
Oct 30 22:51:33.190741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.190627    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46e8ec9f-b22e-4251-9652-118c6b52d401-tls-certs\" (UniqueName: \"kubernetes.io/secret/46e8ec9f-b22e-4251-9652-118c6b52d401-tls-certs\") pod \"46e8ec9f-b22e-4251-9652-118c6b52d401\" (UID: \"46e8ec9f-b22e-4251-9652-118c6b52d401\") "
Oct 30 22:51:33.190741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.190668    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-model-cache" pod "46e8ec9f-b22e-4251-9652-118c6b52d401" (UID: "46e8ec9f-b22e-4251-9652-118c6b52d401"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:33.190925 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.190898    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-home" pod "46e8ec9f-b22e-4251-9652-118c6b52d401" (UID: "46e8ec9f-b22e-4251-9652-118c6b52d401"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:33.191401 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.191371    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:51:33.191401 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.191403    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:51:33.191609 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.191490    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-tmp-dir" pod "46e8ec9f-b22e-4251-9652-118c6b52d401" (UID: "46e8ec9f-b22e-4251-9652-118c6b52d401"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:33.193144 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.193108    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-dshm" pod "46e8ec9f-b22e-4251-9652-118c6b52d401" (UID: "46e8ec9f-b22e-4251-9652-118c6b52d401"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:33.193321 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.193211    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46e8ec9f-b22e-4251-9652-118c6b52d401-tls-certs" pod "46e8ec9f-b22e-4251-9652-118c6b52d401" (UID: "46e8ec9f-b22e-4251-9652-118c6b52d401"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:51:33.193446 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.193421    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e8ec9f-b22e-4251-9652-118c6b52d401-kube-api-access-djvv9" pod "46e8ec9f-b22e-4251-9652-118c6b52d401" (UID: "46e8ec9f-b22e-4251-9652-118c6b52d401"). InnerVolumeSpecName "kube-api-access-djvv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:33.292537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.292493    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-credential-socket\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.292537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.292539    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-workload-certs\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.292801 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.292749    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-istio-envoy\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.292801 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.292788    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014efbd4-b548-4bf9-8b36-50285f54073f-istiod-ca-cert\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.292872 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.292810    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-istio-data\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.292908 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.292895    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014efbd4-b548-4bf9-8b36-50285f54073f-istio-podinfo\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.292944 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.292924    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014efbd4-b548-4bf9-8b36-50285f54073f-istio-token\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.292982 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.292940    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcqjd\" (UniqueName: \"kubernetes.io/projected/014efbd4-b548-4bf9-8b36-50285f54073f-kube-api-access-wcqjd\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.293121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.293094    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-workload-socket\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.293288 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.293273    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:51:33.293340 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.293295    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46e8ec9f-b22e-4251-9652-118c6b52d401-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:51:33.293340 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.293311    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46e8ec9f-b22e-4251-9652-118c6b52d401-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:51:33.293340 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.293321    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djvv9\" (UniqueName: \"kubernetes.io/projected/46e8ec9f-b22e-4251-9652-118c6b52d401-kube-api-access-djvv9\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:51:33.393283 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.393216    4889 generic.go:356] "Generic (PLEG): container finished" podID="46e8ec9f-b22e-4251-9652-118c6b52d401" containerID="f63015797021c1648ab18d9eb2dccd7aa32abfd7dacb318ceb380da7f1408262" exitCode=0
Oct 30 22:51:33.393493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.393288    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks" event={"ID":"46e8ec9f-b22e-4251-9652-118c6b52d401","Type":"ContainerDied","Data":"f63015797021c1648ab18d9eb2dccd7aa32abfd7dacb318ceb380da7f1408262"}
Oct 30 22:51:33.393493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.393342    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks" event={"ID":"46e8ec9f-b22e-4251-9652-118c6b52d401","Type":"ContainerDied","Data":"1f87aa62e4617af5ea1b9ded9f8d3dd8436d78305bc5ca0ae4091826fd4f301b"}
Oct 30 22:51:33.393493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.393365    4889 scope.go:122] "RemoveContainer" containerID="f63015797021c1648ab18d9eb2dccd7aa32abfd7dacb318ceb380da7f1408262"
Oct 30 22:51:33.393493 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.393306    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"
Oct 30 22:51:33.394293 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.394267    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-istio-envoy\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.394446 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.394313    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014efbd4-b548-4bf9-8b36-50285f54073f-istiod-ca-cert\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.394446 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.394335    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-istio-data\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.394446 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.394383    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014efbd4-b548-4bf9-8b36-50285f54073f-istio-podinfo\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.394446 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.394416    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014efbd4-b548-4bf9-8b36-50285f54073f-istio-token\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.394446 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.394441    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcqjd\" (UniqueName: \"kubernetes.io/projected/014efbd4-b548-4bf9-8b36-50285f54073f-kube-api-access-wcqjd\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.394733 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.394488    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-workload-socket\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.394733 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.394588    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-credential-socket\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.394733 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.394613    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-workload-certs\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.395008 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.394984    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-workload-socket\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.395089 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.395067    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-istio-data\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.395154 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.395141    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-workload-certs\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.395310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.395241    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014efbd4-b548-4bf9-8b36-50285f54073f-istiod-ca-cert\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.395405 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.395306    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-credential-socket\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.396970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.396947    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-istio-envoy\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.397301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.397282    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014efbd4-b548-4bf9-8b36-50285f54073f-istio-podinfo\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.407683 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.407661    4889 scope.go:122] "RemoveContainer" containerID="f63015797021c1648ab18d9eb2dccd7aa32abfd7dacb318ceb380da7f1408262"
Oct 30 22:51:33.408039 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:51:33.408019    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f63015797021c1648ab18d9eb2dccd7aa32abfd7dacb318ceb380da7f1408262\": container with ID starting with f63015797021c1648ab18d9eb2dccd7aa32abfd7dacb318ceb380da7f1408262 not found: ID does not exist" containerID="f63015797021c1648ab18d9eb2dccd7aa32abfd7dacb318ceb380da7f1408262"
Oct 30 22:51:33.408123 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.408051    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f63015797021c1648ab18d9eb2dccd7aa32abfd7dacb318ceb380da7f1408262"} err="failed to get container status \"f63015797021c1648ab18d9eb2dccd7aa32abfd7dacb318ceb380da7f1408262\": rpc error: code = NotFound desc = could not find container \"f63015797021c1648ab18d9eb2dccd7aa32abfd7dacb318ceb380da7f1408262\": container with ID starting with f63015797021c1648ab18d9eb2dccd7aa32abfd7dacb318ceb380da7f1408262 not found: ID does not exist"
Oct 30 22:51:33.417663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.417626    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014efbd4-b548-4bf9-8b36-50285f54073f-istio-token\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.417828 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.417768    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcqjd\" (UniqueName: \"kubernetes.io/projected/014efbd4-b548-4bf9-8b36-50285f54073f-kube-api-access-wcqjd\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.445258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.445206    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:33.455380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.455348    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"]
Oct 30 22:51:33.461006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.460844    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-d9f1c2/auth-test-model-d9f1c2-kserve-684c45c4fc-2rsks"]
Oct 30 22:51:33.803498 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.796833    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"]
Oct 30 22:51:33.803498 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:51:33.798470    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod014efbd4_b548_4bf9_8b36_50285f54073f.slice/crio-f3628330c1818c2a6b21102a870c67d4a6fbf2c7082fc39b39dd9edb96292fc8 WatchSource:0}: Error finding container f3628330c1818c2a6b21102a870c67d4a6fbf2c7082fc39b39dd9edb96292fc8: Status 404 returned error can't find the container with id f3628330c1818c2a6b21102a870c67d4a6fbf2c7082fc39b39dd9edb96292fc8
Oct 30 22:51:33.803965 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.803857    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:51:33.804089 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.803964    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:51:33.804089 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.804015    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:51:33.826874 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:33.826835    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:34.073215 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:34.072981    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:34.413084 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:34.412974    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292" event={"ID":"014efbd4-b548-4bf9-8b36-50285f54073f","Type":"ContainerStarted","Data":"15f7d9d87e68b2cb5758593cc553a87bffcfd33f72749e377ae635222f7d6bf7"}
Oct 30 22:51:34.413084 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:34.413037    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292" event={"ID":"014efbd4-b548-4bf9-8b36-50285f54073f","Type":"ContainerStarted","Data":"f3628330c1818c2a6b21102a870c67d4a6fbf2c7082fc39b39dd9edb96292fc8"}
Oct 30 22:51:34.450254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:34.449351    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:34.451262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:34.450635    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 22:51:34.451262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:34.450708    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292" podUID="014efbd4-b548-4bf9-8b36-50285f54073f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 22:51:34.485257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:34.485181    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292" podStartSLOduration=1.485164 podStartE2EDuration="1.485164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:34.475672096 +0000 UTC m=+2679.832038410" watchObservedRunningTime="2026-10-30 22:51:34.485164 +0000 UTC m=+2679.841530314"
Oct 30 22:51:34.539251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:34.532860    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:34.777643 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:34.777606    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46e8ec9f-b22e-4251-9652-118c6b52d401" path="/var/lib/kubelet/pods/46e8ec9f-b22e-4251-9652-118c6b52d401/volumes"
Oct 30 22:51:34.857698 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:34.857663    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"]
Oct 30 22:51:34.864294 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:34.864263    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:34.881553 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:34.881509    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"]
Oct 30 22:51:34.889792 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:34.889755    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-e14e00-openshift-default-dockercfg-sf5l9\""
Oct 30 22:51:34.946381 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:34.946344    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-token\") pod \"e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") " pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:34.946661 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:34.946420    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-data\") pod \"e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") " pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:34.946661 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:34.946506    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-workload-certs\") pod \"e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") " pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:34.946661 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:34.946544    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-credential-socket\") pod \"e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") " pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:34.946661 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:34.946602    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-envoy\") pod \"e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") " pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:34.946661 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:34.946641    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-podinfo\") pod \"e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") " pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:34.947121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:34.946664    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e86e2368-d03b-4634-8245-eddf9d83e16f-istiod-ca-cert\") pod \"e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") " pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:34.947121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:34.946690    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-workload-socket\") pod \"e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") " pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:34.947121 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:34.946746    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49l74\" (UniqueName: \"kubernetes.io/projected/e86e2368-d03b-4634-8245-eddf9d83e16f-kube-api-access-49l74\") pod \"e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") " pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:34.985523 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:34.985469    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:35.047976 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:35.047881    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-envoy\") pod \"e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") " pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:35.047976 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:35.047963    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-podinfo\") pod \"e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") " pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:35.048273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:35.048013    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e86e2368-d03b-4634-8245-eddf9d83e16f-istiod-ca-cert\") pod \"e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") " pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:35.048273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:35.048060    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-workload-socket\") pod \"e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") " pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:35.048273 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:35.048156    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49l74\" (UniqueName: \"kubernetes.io/projected/e86e2368-d03b-4634-8245-eddf9d83e16f-kube-api-access-49l74\") pod \"e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") " pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:35.048447 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:35.048352    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-token\") pod \"e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") " pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:35.048447 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:35.048423    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-data\") pod \"e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") " pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:35.048562 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:35.048475    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-workload-certs\") pod \"e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") " pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:35.048562 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:35.048516    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-credential-socket\") pod \"e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") " pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:35.048562 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:35.048544    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-workload-socket\") pod \"e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") " pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:35.048846 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:35.048813    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-data\") pod \"e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") " pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:35.048974 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:35.048861    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e86e2368-d03b-4634-8245-eddf9d83e16f-istiod-ca-cert\") pod \"e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") " pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:35.048974 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:35.048895    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-credential-socket\") pod \"e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") " pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:35.048974 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:35.048924    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-workload-certs\") pod \"e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") " pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:35.050928 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:35.050899    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-envoy\") pod \"e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") " pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:35.051046 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:35.050971    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-podinfo\") pod \"e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") " pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:35.084246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:35.083506    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49l74\" (UniqueName: \"kubernetes.io/projected/e86e2368-d03b-4634-8245-eddf9d83e16f-kube-api-access-49l74\") pod \"e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") " pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:35.084246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:35.083775    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-token\") pod \"e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") " pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:35.189250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:35.188019    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:35.189250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:35.188869    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:35.448288 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:35.447497    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 22:51:35.448288 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:35.447554    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292" podUID="014efbd4-b548-4bf9-8b36-50285f54073f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 22:51:35.570384 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:51:35.570350    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86e2368_d03b_4634_8245_eddf9d83e16f.slice/crio-adbdb77d559a01dc215ae4f6a2b25fd80795906954ef2c2bd757339ae41cfdfa WatchSource:0}: Error finding container adbdb77d559a01dc215ae4f6a2b25fd80795906954ef2c2bd757339ae41cfdfa: Status 404 returned error can't find the container with id adbdb77d559a01dc215ae4f6a2b25fd80795906954ef2c2bd757339ae41cfdfa
Oct 30 22:51:35.571826 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:35.571797    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"]
Oct 30 22:51:35.574021 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:35.573984    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:51:35.574103 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:35.574058    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:51:35.574103 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:35.574088    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:51:35.738247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:35.731620    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:36.341808 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:36.341779    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc"
Oct 30 22:51:36.435290 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:36.431804    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"
Oct 30 22:51:36.446202 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:36.446160    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 22:51:36.446439 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:36.446266    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292" podUID="014efbd4-b548-4bf9-8b36-50285f54073f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 22:51:36.458453 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:36.458425    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"
Oct 30 22:51:36.467064 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:36.467028    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k" event={"ID":"e86e2368-d03b-4634-8245-eddf9d83e16f","Type":"ContainerStarted","Data":"dd26b8faf73cb1e70647d9b1340ea734bcba1abca88c079ade95ec959d956c59"}
Oct 30 22:51:36.467291 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:36.467083    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k" event={"ID":"e86e2368-d03b-4634-8245-eddf9d83e16f","Type":"ContainerStarted","Data":"adbdb77d559a01dc215ae4f6a2b25fd80795906954ef2c2bd757339ae41cfdfa"}
Oct 30 22:51:36.482613 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:36.482586    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"
Oct 30 22:51:36.607618 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:36.607471    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k" podStartSLOduration=2.607448058 podStartE2EDuration="2.607448058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:36.605091156 +0000 UTC m=+2681.961457471" watchObservedRunningTime="2026-10-30 22:51:36.607448058 +0000 UTC m=+2681.963814373"
Oct 30 22:51:37.189255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:37.189198    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:37.198043 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:37.198014    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:37.445758 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:37.445657    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 22:51:37.445758 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:37.445731    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292" podUID="014efbd4-b548-4bf9-8b36-50285f54073f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 22:51:37.476858 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:37.476811    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:37.478125 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:37.478100    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:51:37.604926 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:37.604844    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:38.446421 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:38.446385    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 22:51:38.446841 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:38.446442    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292" podUID="014efbd4-b548-4bf9-8b36-50285f54073f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 22:51:39.137251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.136972    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:39.292044 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.291999    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"]
Oct 30 22:51:39.301107 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.301072    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.354267 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.345758    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-3cd77f-openshift-default-dockercfg-vlt6j\""
Oct 30 22:51:39.355934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.355679    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"]
Oct 30 22:51:39.446068 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.445977    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 22:51:39.446068 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.446037    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292" podUID="014efbd4-b548-4bf9-8b36-50285f54073f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 22:51:39.465359 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.465312    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65b844a4-133d-40a6-bc49-81bbffc94221-istiod-ca-cert\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.465892 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.465435    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-istio-data\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.465892 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.465572    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-istio-envoy\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.465892 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.465681    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65b844a4-133d-40a6-bc49-81bbffc94221-istio-podinfo\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.465892 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.465740    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-credential-socket\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.465892 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.465826    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65b844a4-133d-40a6-bc49-81bbffc94221-istio-token\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.466114 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.465949    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-workload-socket\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.466114 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.465982    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-workload-certs\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.466114 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.466066    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfdlq\" (UniqueName: \"kubernetes.io/projected/65b844a4-133d-40a6-bc49-81bbffc94221-kube-api-access-nfdlq\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.470902 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.470869    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:39.567053 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.567010    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65b844a4-133d-40a6-bc49-81bbffc94221-istio-podinfo\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.567240 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.567071    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-credential-socket\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.567240 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.567147    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65b844a4-133d-40a6-bc49-81bbffc94221-istio-token\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.567345 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.567312    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-workload-socket\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.567380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.567348    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-workload-certs\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.567421 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.567381    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfdlq\" (UniqueName: \"kubernetes.io/projected/65b844a4-133d-40a6-bc49-81bbffc94221-kube-api-access-nfdlq\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.567571 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.567528    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65b844a4-133d-40a6-bc49-81bbffc94221-istiod-ca-cert\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.567863 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.567565    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-credential-socket\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.567863 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.567672    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-istio-data\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.567863 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.567756    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-istio-envoy\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.568053 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.568005    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-workload-socket\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.568120 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.568093    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65b844a4-133d-40a6-bc49-81bbffc94221-istiod-ca-cert\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.568240 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.568197    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-istio-data\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.568321 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.568280    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-workload-certs\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.570567 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.570509    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-istio-envoy\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.570884 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.570848    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65b844a4-133d-40a6-bc49-81bbffc94221-istio-podinfo\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.590391 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.590270    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfdlq\" (UniqueName: \"kubernetes.io/projected/65b844a4-133d-40a6-bc49-81bbffc94221-kube-api-access-nfdlq\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.590571 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.590395    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65b844a4-133d-40a6-bc49-81bbffc94221-istio-token\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.651741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.651700    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:39.733427 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:39.733140    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:40.032465 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:40.030056    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"]
Oct 30 22:51:40.036415 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:40.036377    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:51:40.036537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:40.036482    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:51:40.036537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:40.036511    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:51:40.306668 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:51:40.306543    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:51:40.505297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:40.505262    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp" event={"ID":"65b844a4-133d-40a6-bc49-81bbffc94221","Type":"ContainerStarted","Data":"6cbde591fc250ce6ef68b813f77a7c02533d36d7bb56a89dd70ce9992f5eedc8"}
Oct 30 22:51:40.505860 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:40.505828    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp" event={"ID":"65b844a4-133d-40a6-bc49-81bbffc94221","Type":"ContainerStarted","Data":"d7c8f696bed537c96e3fefee649fc8b5444e2edd83c2cc3ab83ddb957ac777d1"}
Oct 30 22:51:40.562274 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:40.562136    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp" podStartSLOduration=1.562113307 podStartE2EDuration="1.562113307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:40.555705152 +0000 UTC m=+2685.912071494" watchObservedRunningTime="2026-10-30 22:51:40.562113307 +0000 UTC m=+2685.918479623"
Oct 30 22:51:40.654517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:40.654466    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:40.700682 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:40.700450    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:40.701397 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:40.701370    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:40.702707 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:40.702687    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:51:40.906856 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:40.906759    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:41.093051 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:41.093014    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:41.653572 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:41.653502    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:41.654678 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:41.654649    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 30 22:51:41.654882 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:41.654707    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp" podUID="65b844a4-133d-40a6-bc49-81bbffc94221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 30 22:51:42.652504 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:42.652464    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 30 22:51:42.652696 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:42.652525    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp" podUID="65b844a4-133d-40a6-bc49-81bbffc94221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 30 22:51:43.392007 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:43.391969    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:43.653113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:43.653026    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 30 22:51:43.653113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:43.653081    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp" podUID="65b844a4-133d-40a6-bc49-81bbffc94221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 30 22:51:44.652583 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:44.652541    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 30 22:51:44.653036 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:44.652602    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp" podUID="65b844a4-133d-40a6-bc49-81bbffc94221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 30 22:51:45.155758 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:45.155721    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:45.653186 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:45.653148    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 30 22:51:45.653703 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:45.653216    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp" podUID="65b844a4-133d-40a6-bc49-81bbffc94221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 30 22:51:46.652875 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:46.652830    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 30 22:51:46.653079 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:46.652907    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp" podUID="65b844a4-133d-40a6-bc49-81bbffc94221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 30 22:51:46.837246 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:46.834366    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:47.659206 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:47.659174    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:47.659619 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:47.659599    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:47.660549 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:47.660528    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 22:51:48.713057 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:48.713021    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:50.200829 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:50.200788    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:50.507848 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:50.507806    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:52.344671 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:52.344637    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:54.036811 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:54.036773    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:55.668427 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:55.668364    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 22:51:55.672055 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:55.672031    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 22:51:55.738825 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:55.738781    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:57.564779 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:57.564735    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:59.246064 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:59.246020    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:59.409683 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:51:59.409648    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:01.225002 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:01.224968    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:01.401389 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:01.401353    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:03.064468 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:03.064430    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:04.825942 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:04.825906    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:06.698606 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:06.698562    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:08.434652 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:08.434611    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:10.156639 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:10.156589    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:10.306070 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:52:10.306033    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:52:11.896237 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:11.896141    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:13.531677 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.531636    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"]
Oct 30 22:52:13.538294 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.538260    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.560953 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.560912    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"]
Oct 30 22:52:13.616393 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.616342    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-cdc122-openshift-default-dockercfg-4sxwk\""
Oct 30 22:52:13.636343 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.636303    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:13.666047 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.666010    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istiod-ca-cert\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.666274 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.666181    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-workload-socket\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.666351 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.666329    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-token\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.666405 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.666389    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-credential-socket\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.666530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.666509    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf9xf\" (UniqueName: \"kubernetes.io/projected/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-kube-api-access-mf9xf\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.666587 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.666571    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-data\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.666651 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.666632    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-podinfo\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.666695 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.666670    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-workload-certs\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.666695 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.666686    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-envoy\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.767546 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.767508    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-token\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.767750 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.767557    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-credential-socket\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.767750 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.767588    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf9xf\" (UniqueName: \"kubernetes.io/projected/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-kube-api-access-mf9xf\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.767750 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.767613    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-data\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.767750 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.767635    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-podinfo\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.767750 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.767657    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-workload-certs\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.767750 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.767673    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-envoy\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.768103 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.767767    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istiod-ca-cert\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.768103 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.767864    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-workload-socket\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.768103 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.768031    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-credential-socket\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.768298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.768114    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-workload-certs\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.768298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.768168    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-data\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.768298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.768278    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-workload-socket\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.768508 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.768490    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istiod-ca-cert\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.770466 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.770435    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-envoy\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.770637 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.770620    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-podinfo\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.789311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.789166    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-token\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.789311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.789290    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf9xf\" (UniqueName: \"kubernetes.io/projected/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-kube-api-access-mf9xf\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:13.854693 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:13.854650    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:14.191704 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:52:14.191670    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24e8fc0c_97b0_47f1_ba4b_5ec430718d31.slice/crio-4241fe15034384047bed8db225edc71620ba8960f2b9d463554d7ab72ac02acc WatchSource:0}: Error finding container 4241fe15034384047bed8db225edc71620ba8960f2b9d463554d7ab72ac02acc: Status 404 returned error can't find the container with id 4241fe15034384047bed8db225edc71620ba8960f2b9d463554d7ab72ac02acc
Oct 30 22:52:14.192314 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:14.192288    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"]
Oct 30 22:52:14.194744 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:14.194712    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:52:14.194868 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:14.194781    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:52:14.194868 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:14.194815    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:52:14.861729 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:14.861692    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x" event={"ID":"24e8fc0c-97b0-47f1-ba4b-5ec430718d31","Type":"ContainerStarted","Data":"533f76e65b54feab7f5111481ed774d33442f3ac754e6eddc48347cda2f0e85b"}
Oct 30 22:52:14.862173 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:14.861756    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x" event={"ID":"24e8fc0c-97b0-47f1-ba4b-5ec430718d31","Type":"ContainerStarted","Data":"4241fe15034384047bed8db225edc71620ba8960f2b9d463554d7ab72ac02acc"}
Oct 30 22:52:15.452489 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:15.452453    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:15.854809 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:15.854773    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:15.857301 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:15.857273    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Oct 30 22:52:15.857485 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:15.857322    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x" podUID="24e8fc0c-97b0-47f1-ba4b-5ec430718d31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Oct 30 22:52:16.855408 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:16.855366    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Oct 30 22:52:16.855907 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:16.855435    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x" podUID="24e8fc0c-97b0-47f1-ba4b-5ec430718d31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Oct 30 22:52:17.361270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:17.361203    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:17.855991 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:17.855959    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Oct 30 22:52:17.856478 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:17.856025    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x" podUID="24e8fc0c-97b0-47f1-ba4b-5ec430718d31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Oct 30 22:52:18.855923 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:18.855890    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Oct 30 22:52:18.856130 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:18.855945    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x" podUID="24e8fc0c-97b0-47f1-ba4b-5ec430718d31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Oct 30 22:52:19.094671 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:19.094636    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:19.335930 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:19.335881    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:19.504928 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:19.504892    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:19.859609 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:19.859559    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Oct 30 22:52:19.860005 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:19.859673    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x" podUID="24e8fc0c-97b0-47f1-ba4b-5ec430718d31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Oct 30 22:52:20.861748 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:20.861710    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:20.862281 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:20.862263    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:20.862928 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:20.862910    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 22:52:20.928407 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:20.928342    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x" podStartSLOduration=7.928325032 podStartE2EDuration="7.928325032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:14.936675178 +0000 UTC m=+2720.293041494" watchObservedRunningTime="2026-10-30 22:52:20.928325032 +0000 UTC m=+2726.284691347"
Oct 30 22:52:21.047040 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:21.046984    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:22.860075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:22.860039    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:24.612472 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:24.612433    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:26.639050 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:26.639015    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:28.441046 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:28.441010    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:30.203988 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:30.203932    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:31.987613 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:31.987571    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:33.718824 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:33.718768    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:33.892523 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:33.892485    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:35.485558 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:35.485293    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:37.139844 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:37.139812    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:39.010678 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:39.010633    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:40.303740 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:52:40.303696    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:52:40.751497 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:40.751452    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:42.661383 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:42.661345    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:44.107608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:44.107569    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:44.674174 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:44.674132    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:45.012786 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:45.012749    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:45.522711 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:45.522666    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:45.656880 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:45.656702    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc"]
Oct 30 22:52:45.715109 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:45.715073    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:45.964993 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:45.964896    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:46.182141 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:46.182105    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:46.362896 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:46.362855    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:48.110971 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:48.110925    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:49.670927 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:49.670883    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:49.849587 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:49.849546    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:50.121839 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:50.121789    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:50.401535 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:50.401435    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:50.657757 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:50.657626    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc" podUID="879f116a-b7d8-4db8-af80-e76b9018c22f" containerName="payload-pre-processing" containerID="cri-o://48a9771e69e02858520c90113581205bff49e9dca7c3cd9cff19debc8c96330c" gracePeriod=25
Oct 30 22:52:50.958848 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:50.958814    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc"
Oct 30 22:52:51.097686 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.097647    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/879f116a-b7d8-4db8-af80-e76b9018c22f-kube-api-access-tdk6t\" (UniqueName: \"kubernetes.io/projected/879f116a-b7d8-4db8-af80-e76b9018c22f-kube-api-access-tdk6t\") pod \"879f116a-b7d8-4db8-af80-e76b9018c22f\" (UID: \"879f116a-b7d8-4db8-af80-e76b9018c22f\") "
Oct 30 22:52:51.097950 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.097928    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/879f116a-b7d8-4db8-af80-e76b9018c22f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879f116a-b7d8-4db8-af80-e76b9018c22f-plugins-config-volume\") pod \"879f116a-b7d8-4db8-af80-e76b9018c22f\" (UID: \"879f116a-b7d8-4db8-af80-e76b9018c22f\") "
Oct 30 22:52:51.098587 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.098386    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/879f116a-b7d8-4db8-af80-e76b9018c22f-plugins-config-volume" pod "879f116a-b7d8-4db8-af80-e76b9018c22f" (UID: "879f116a-b7d8-4db8-af80-e76b9018c22f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:51.098828 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.098752    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879f116a-b7d8-4db8-af80-e76b9018c22f-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:52:51.101487 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.101456    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/879f116a-b7d8-4db8-af80-e76b9018c22f-kube-api-access-tdk6t" pod "879f116a-b7d8-4db8-af80-e76b9018c22f" (UID: "879f116a-b7d8-4db8-af80-e76b9018c22f"). InnerVolumeSpecName "kube-api-access-tdk6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:51.200044 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.199948    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdk6t\" (UniqueName: \"kubernetes.io/projected/879f116a-b7d8-4db8-af80-e76b9018c22f-kube-api-access-tdk6t\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:52:51.246313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.246280    4889 generic.go:356] "Generic (PLEG): container finished" podID="879f116a-b7d8-4db8-af80-e76b9018c22f" containerID="48a9771e69e02858520c90113581205bff49e9dca7c3cd9cff19debc8c96330c" exitCode=0
Oct 30 22:52:51.246522 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.246321    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc" event={"ID":"879f116a-b7d8-4db8-af80-e76b9018c22f","Type":"ContainerDied","Data":"48a9771e69e02858520c90113581205bff49e9dca7c3cd9cff19debc8c96330c"}
Oct 30 22:52:51.246522 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.246359    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc"
Oct 30 22:52:51.246522 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.246379    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc" event={"ID":"879f116a-b7d8-4db8-af80-e76b9018c22f","Type":"ContainerDied","Data":"e8df9f557ac30b70a99c576540c69b542a7f7c7053a0dd3ce2a592034ce000c2"}
Oct 30 22:52:51.246522 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.246402    4889 scope.go:122] "RemoveContainer" containerID="48a9771e69e02858520c90113581205bff49e9dca7c3cd9cff19debc8c96330c"
Oct 30 22:52:51.258754 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.258731    4889 scope.go:122] "RemoveContainer" containerID="48a9771e69e02858520c90113581205bff49e9dca7c3cd9cff19debc8c96330c"
Oct 30 22:52:51.259079 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:52:51.259057    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48a9771e69e02858520c90113581205bff49e9dca7c3cd9cff19debc8c96330c\": container with ID starting with 48a9771e69e02858520c90113581205bff49e9dca7c3cd9cff19debc8c96330c not found: ID does not exist" containerID="48a9771e69e02858520c90113581205bff49e9dca7c3cd9cff19debc8c96330c"
Oct 30 22:52:51.259167 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.259090    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48a9771e69e02858520c90113581205bff49e9dca7c3cd9cff19debc8c96330c"} err="failed to get container status \"48a9771e69e02858520c90113581205bff49e9dca7c3cd9cff19debc8c96330c\": rpc error: code = NotFound desc = could not find container \"48a9771e69e02858520c90113581205bff49e9dca7c3cd9cff19debc8c96330c\": container with ID starting with 48a9771e69e02858520c90113581205bff49e9dca7c3cd9cff19debc8c96330c not found: ID does not exist"
Oct 30 22:52:51.347714 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.347676    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc"]
Oct 30 22:52:51.361907 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.361873    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-55c66888c9-fbrpc"]
Oct 30 22:52:51.490067 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.489971    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:51.601053 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.601014    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68"]
Oct 30 22:52:51.602030 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.602004    4889 cpu_manager.go:414] "containerMap: removing container" podUID="879f116a-b7d8-4db8-af80-e76b9018c22f" containerName="payload-pre-processing"
Oct 30 22:52:51.602030 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.602028    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="879f116a-b7d8-4db8-af80-e76b9018c22f" containerName="payload-pre-processing"
Oct 30 22:52:51.602255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.602158    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="879f116a-b7d8-4db8-af80-e76b9018c22f" containerName="payload-pre-processing"
Oct 30 22:52:51.607932 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.607907    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68"
Oct 30 22:52:51.624130 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.624095    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68"]
Oct 30 22:52:51.633081 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.633047    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a60e80-dockercfg-7hmzd\""
Oct 30 22:52:51.638948 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.638915    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a60e80\""
Oct 30 22:52:51.655353 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.655311    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m"]
Oct 30 22:52:51.661583 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.661542    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m"
Oct 30 22:52:51.732238 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.732187    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2006c00-f096-4380-b39e-09c52d3f7ee0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68\" (UID: \"b2006c00-f096-4380-b39e-09c52d3f7ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68"
Oct 30 22:52:51.732531 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.732507    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhtzz\" (UniqueName: \"kubernetes.io/projected/b2006c00-f096-4380-b39e-09c52d3f7ee0-kube-api-access-fhtzz\") pod \"payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68\" (UID: \"b2006c00-f096-4380-b39e-09c52d3f7ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68"
Oct 30 22:52:51.814753 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.795085    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m"]
Oct 30 22:52:51.814753 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.802867    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:51.834855 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.834813    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99xbn\" (UniqueName: \"kubernetes.io/projected/53046f1e-ac05-4261-8258-b6109ecdea33-kube-api-access-99xbn\") pod \"payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m\" (UID: \"53046f1e-ac05-4261-8258-b6109ecdea33\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m"
Oct 30 22:52:51.835034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.835023    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53046f1e-ac05-4261-8258-b6109ecdea33-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m\" (UID: \"53046f1e-ac05-4261-8258-b6109ecdea33\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m"
Oct 30 22:52:51.836139 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.835215    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2006c00-f096-4380-b39e-09c52d3f7ee0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68\" (UID: \"b2006c00-f096-4380-b39e-09c52d3f7ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68"
Oct 30 22:52:51.836139 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.835292    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhtzz\" (UniqueName: \"kubernetes.io/projected/b2006c00-f096-4380-b39e-09c52d3f7ee0-kube-api-access-fhtzz\") pod \"payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68\" (UID: \"b2006c00-f096-4380-b39e-09c52d3f7ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68"
Oct 30 22:52:51.836139 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.836091    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2006c00-f096-4380-b39e-09c52d3f7ee0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68\" (UID: \"b2006c00-f096-4380-b39e-09c52d3f7ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68"
Oct 30 22:52:51.858354 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.858319    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhtzz\" (UniqueName: \"kubernetes.io/projected/b2006c00-f096-4380-b39e-09c52d3f7ee0-kube-api-access-fhtzz\") pod \"payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68\" (UID: \"b2006c00-f096-4380-b39e-09c52d3f7ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68"
Oct 30 22:52:51.934443 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.934404    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68"
Oct 30 22:52:51.937200 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.937167    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99xbn\" (UniqueName: \"kubernetes.io/projected/53046f1e-ac05-4261-8258-b6109ecdea33-kube-api-access-99xbn\") pod \"payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m\" (UID: \"53046f1e-ac05-4261-8258-b6109ecdea33\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m"
Oct 30 22:52:51.937368 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.937278    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53046f1e-ac05-4261-8258-b6109ecdea33-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m\" (UID: \"53046f1e-ac05-4261-8258-b6109ecdea33\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m"
Oct 30 22:52:51.938091 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.938024    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53046f1e-ac05-4261-8258-b6109ecdea33-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m\" (UID: \"53046f1e-ac05-4261-8258-b6109ecdea33\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m"
Oct 30 22:52:51.960880 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:51.960848    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99xbn\" (UniqueName: \"kubernetes.io/projected/53046f1e-ac05-4261-8258-b6109ecdea33-kube-api-access-99xbn\") pod \"payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m\" (UID: \"53046f1e-ac05-4261-8258-b6109ecdea33\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m"
Oct 30 22:52:52.028604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:52.028547    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:52.108167 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:52.108072    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m"
Oct 30 22:52:52.223639 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:52.223599    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:52.254542 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:52.253935    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68"]
Oct 30 22:52:52.258397 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:52:52.258364    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2006c00_f096_4380_b39e_09c52d3f7ee0.slice/crio-1d1bdfd8f3276520451ee1bf3f1dfdf5c5d6e00f0c04678639ecbe7c8d138cc7 WatchSource:0}: Error finding container 1d1bdfd8f3276520451ee1bf3f1dfdf5c5d6e00f0c04678639ecbe7c8d138cc7: Status 404 returned error can't find the container with id 1d1bdfd8f3276520451ee1bf3f1dfdf5c5d6e00f0c04678639ecbe7c8d138cc7
Oct 30 22:52:52.423591 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:52:52.423551    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53046f1e_ac05_4261_8258_b6109ecdea33.slice/crio-163ca8219cf78036f8734444c5a3f3b3429ebe7b10b02a91896ed1535c55868b WatchSource:0}: Error finding container 163ca8219cf78036f8734444c5a3f3b3429ebe7b10b02a91896ed1535c55868b: Status 404 returned error can't find the container with id 163ca8219cf78036f8734444c5a3f3b3429ebe7b10b02a91896ed1535c55868b
Oct 30 22:52:52.424535 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:52.424509    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m"]
Oct 30 22:52:52.777291 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:52.777258    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="879f116a-b7d8-4db8-af80-e76b9018c22f" path="/var/lib/kubelet/pods/879f116a-b7d8-4db8-af80-e76b9018c22f/volumes"
Oct 30 22:52:53.276582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:53.276542    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m" event={"ID":"53046f1e-ac05-4261-8258-b6109ecdea33","Type":"ContainerStarted","Data":"84f634ce947a3652a4dddd9e7733ee5ccc91493a08b5ec82c57f1b04bfc3c0a4"}
Oct 30 22:52:53.277138 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:53.276618    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m"
Oct 30 22:52:53.277138 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:53.276642    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m" event={"ID":"53046f1e-ac05-4261-8258-b6109ecdea33","Type":"ContainerStarted","Data":"163ca8219cf78036f8734444c5a3f3b3429ebe7b10b02a91896ed1535c55868b"}
Oct 30 22:52:53.278409 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:53.278382    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68" event={"ID":"b2006c00-f096-4380-b39e-09c52d3f7ee0","Type":"ContainerStarted","Data":"a0b4e53ca0cc2e7cf8e70759536bed5fcdc5a985893609bb1f827732f12a751d"}
Oct 30 22:52:53.278576 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:53.278424    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68" event={"ID":"b2006c00-f096-4380-b39e-09c52d3f7ee0","Type":"ContainerStarted","Data":"1d1bdfd8f3276520451ee1bf3f1dfdf5c5d6e00f0c04678639ecbe7c8d138cc7"}
Oct 30 22:52:53.278576 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:53.278508    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68"
Oct 30 22:52:53.331651 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:53.331582    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m" podStartSLOduration=2.33156809 podStartE2EDuration="2.33156809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:53.329558282 +0000 UTC m=+2758.685924596" watchObservedRunningTime="2026-10-30 22:52:53.33156809 +0000 UTC m=+2758.687934405"
Oct 30 22:52:53.373012 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:53.372952    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68" podStartSLOduration=2.372936877 podStartE2EDuration="2.372936877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:53.371613612 +0000 UTC m=+2758.727979929" watchObservedRunningTime="2026-10-30 22:52:53.372936877 +0000 UTC m=+2758.729303263"
Oct 30 22:52:53.927454 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:53.927411    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:55.917694 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:55.917647    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:57.652335 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:57.652295    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:59.430361 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:52:59.430324    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:01.219400 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:01.219358    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:02.918935 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:02.918896    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:04.290831 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:04.290799    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68"
Oct 30 22:53:04.291278 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:04.290886    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m"
Oct 30 22:53:04.808810 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:04.808774    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:05.914571 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:05.914532    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:06.543602 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:06.543567    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:07.998464 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:07.998423    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"]
Oct 30 22:53:08.004777 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.004750    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"
Oct 30 22:53:08.032571 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.032518    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"]
Oct 30 22:53:08.033681 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.033645    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-cdc122\"/\"default-dockercfg-zbbz9\""
Oct 30 22:53:08.033919 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.033696    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-cdc122\"/\"rate-test-model-cdc122-kserve-self-signed-certs\""
Oct 30 22:53:08.034398 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.034375    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-cdc122\"/\"openshift-service-ca.crt\""
Oct 30 22:53:08.034640 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.034619    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-cdc122\"/\"kube-root-ca.crt\""
Oct 30 22:53:08.106761 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.106714    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d19c99b8-8347-4314-83f8-7cb17634b961-tls-certs\") pod \"rate-test-model-cdc122-kserve-d5766965-p6tnz\" (UID: \"d19c99b8-8347-4314-83f8-7cb17634b961\") " pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"
Oct 30 22:53:08.106929 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.106836    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-model-cache\") pod \"rate-test-model-cdc122-kserve-d5766965-p6tnz\" (UID: \"d19c99b8-8347-4314-83f8-7cb17634b961\") " pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"
Oct 30 22:53:08.106929 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.106891    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-dshm\") pod \"rate-test-model-cdc122-kserve-d5766965-p6tnz\" (UID: \"d19c99b8-8347-4314-83f8-7cb17634b961\") " pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"
Oct 30 22:53:08.107030 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.106931    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-tmp-dir\") pod \"rate-test-model-cdc122-kserve-d5766965-p6tnz\" (UID: \"d19c99b8-8347-4314-83f8-7cb17634b961\") " pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"
Oct 30 22:53:08.107175 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.107152    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-home\") pod \"rate-test-model-cdc122-kserve-d5766965-p6tnz\" (UID: \"d19c99b8-8347-4314-83f8-7cb17634b961\") " pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"
Oct 30 22:53:08.107240 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.107201    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqvxs\" (UniqueName: \"kubernetes.io/projected/d19c99b8-8347-4314-83f8-7cb17634b961-kube-api-access-cqvxs\") pod \"rate-test-model-cdc122-kserve-d5766965-p6tnz\" (UID: \"d19c99b8-8347-4314-83f8-7cb17634b961\") " pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"
Oct 30 22:53:08.117134 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.117103    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:08.208700 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.208638    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d19c99b8-8347-4314-83f8-7cb17634b961-tls-certs\") pod \"rate-test-model-cdc122-kserve-d5766965-p6tnz\" (UID: \"d19c99b8-8347-4314-83f8-7cb17634b961\") " pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"
Oct 30 22:53:08.208910 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.208717    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-model-cache\") pod \"rate-test-model-cdc122-kserve-d5766965-p6tnz\" (UID: \"d19c99b8-8347-4314-83f8-7cb17634b961\") " pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"
Oct 30 22:53:08.208910 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.208757    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-dshm\") pod \"rate-test-model-cdc122-kserve-d5766965-p6tnz\" (UID: \"d19c99b8-8347-4314-83f8-7cb17634b961\") " pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"
Oct 30 22:53:08.208910 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.208795    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-tmp-dir\") pod \"rate-test-model-cdc122-kserve-d5766965-p6tnz\" (UID: \"d19c99b8-8347-4314-83f8-7cb17634b961\") " pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"
Oct 30 22:53:08.209157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.208953    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-home\") pod \"rate-test-model-cdc122-kserve-d5766965-p6tnz\" (UID: \"d19c99b8-8347-4314-83f8-7cb17634b961\") " pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"
Oct 30 22:53:08.209157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.208986    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqvxs\" (UniqueName: \"kubernetes.io/projected/d19c99b8-8347-4314-83f8-7cb17634b961-kube-api-access-cqvxs\") pod \"rate-test-model-cdc122-kserve-d5766965-p6tnz\" (UID: \"d19c99b8-8347-4314-83f8-7cb17634b961\") " pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"
Oct 30 22:53:08.209344 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.209206    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-model-cache\") pod \"rate-test-model-cdc122-kserve-d5766965-p6tnz\" (UID: \"d19c99b8-8347-4314-83f8-7cb17634b961\") " pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"
Oct 30 22:53:08.209344 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.209323    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-tmp-dir\") pod \"rate-test-model-cdc122-kserve-d5766965-p6tnz\" (UID: \"d19c99b8-8347-4314-83f8-7cb17634b961\") " pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"
Oct 30 22:53:08.209439 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.209409    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-home\") pod \"rate-test-model-cdc122-kserve-d5766965-p6tnz\" (UID: \"d19c99b8-8347-4314-83f8-7cb17634b961\") " pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"
Oct 30 22:53:08.211379 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.211358    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-dshm\") pod \"rate-test-model-cdc122-kserve-d5766965-p6tnz\" (UID: \"d19c99b8-8347-4314-83f8-7cb17634b961\") " pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"
Oct 30 22:53:08.211660 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.211640    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d19c99b8-8347-4314-83f8-7cb17634b961-tls-certs\") pod \"rate-test-model-cdc122-kserve-d5766965-p6tnz\" (UID: \"d19c99b8-8347-4314-83f8-7cb17634b961\") " pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"
Oct 30 22:53:08.230639 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.230600    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqvxs\" (UniqueName: \"kubernetes.io/projected/d19c99b8-8347-4314-83f8-7cb17634b961-kube-api-access-cqvxs\") pod \"rate-test-model-cdc122-kserve-d5766965-p6tnz\" (UID: \"d19c99b8-8347-4314-83f8-7cb17634b961\") " pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"
Oct 30 22:53:08.323284 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.323175    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"
Oct 30 22:53:08.431923 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.431881    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:08.633897 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:53:08.633861    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd19c99b8_8347_4314_83f8_7cb17634b961.slice/crio-be5d31bde5e1047b2140d9f503d0cc9e2739483235da5fc6d04dbd871e198bbc WatchSource:0}: Error finding container be5d31bde5e1047b2140d9f503d0cc9e2739483235da5fc6d04dbd871e198bbc: Status 404 returned error can't find the container with id be5d31bde5e1047b2140d9f503d0cc9e2739483235da5fc6d04dbd871e198bbc
Oct 30 22:53:08.634600 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:08.634571    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"]
Oct 30 22:53:09.440808 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:09.440766    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz" event={"ID":"d19c99b8-8347-4314-83f8-7cb17634b961","Type":"ContainerStarted","Data":"e578897eec8571c8fc848017af675c7c6e61b36b70e16cce47b79750f93d5fdf"}
Oct 30 22:53:09.440808 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:09.440822    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz" event={"ID":"d19c99b8-8347-4314-83f8-7cb17634b961","Type":"ContainerStarted","Data":"be5d31bde5e1047b2140d9f503d0cc9e2739483235da5fc6d04dbd871e198bbc"}
Oct 30 22:53:09.486906 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:09.486849    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz" podStartSLOduration=2.486833113 podStartE2EDuration="2.486833113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:53:09.482490574 +0000 UTC m=+2774.838856889" watchObservedRunningTime="2026-10-30 22:53:09.486833113 +0000 UTC m=+2774.843199426"
Oct 30 22:53:10.120830 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:10.120789    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:10.305679 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:53:10.305641    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:53:10.323914 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:10.323883    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"
Oct 30 22:53:11.735932 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:11.735886    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:13.584507 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:13.584472    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:15.211043 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:15.211007    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:17.062130 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:17.062081    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:18.323746 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:18.323712    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"
Oct 30 22:53:18.336626 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:18.336587    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"
Oct 30 22:53:18.545463 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:18.545428    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"
Oct 30 22:53:18.742298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:18.742261    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:20.423173 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:20.423128    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:22.186575 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:22.186528    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:23.445764 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:23.445723    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:23.631719 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:23.631666    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:23.814383 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:23.814349    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:24.609485 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:24.609445    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:26.361865 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:26.361822    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:28.196774 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:28.196735    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:29.941598 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:29.941551    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:31.658502 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:31.658466    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:31.858033 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:31.857993    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:33.561000 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:33.560955    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:35.634908 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:35.634870    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:37.736262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:37.735533    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:39.631765 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:39.631662    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:40.308477 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:53:40.308423    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:53:41.340692 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:41.340481    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:43.017648 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:43.017598    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:43.615437 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:43.615402    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:43.791276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:43.791143    4889 generic.go:356] "Generic (PLEG): container finished" podID="e86e2368-d03b-4634-8245-eddf9d83e16f" containerID="dd26b8faf73cb1e70647d9b1340ea734bcba1abca88c079ade95ec959d956c59" exitCode=0
Oct 30 22:53:43.791276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:43.791188    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k" event={"ID":"e86e2368-d03b-4634-8245-eddf9d83e16f","Type":"ContainerDied","Data":"dd26b8faf73cb1e70647d9b1340ea734bcba1abca88c079ade95ec959d956c59"}
Oct 30 22:53:43.791777 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:43.791757    4889 scope.go:122] "RemoveContainer" containerID="dd26b8faf73cb1e70647d9b1340ea734bcba1abca88c079ade95ec959d956c59"
Oct 30 22:53:43.794013 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:43.793981    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:53:43.794125 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:43.794058    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:53:43.794125 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:43.794091    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:53:44.743279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:44.742467    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:44.804995 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:44.804956    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k" event={"ID":"e86e2368-d03b-4634-8245-eddf9d83e16f","Type":"ContainerStarted","Data":"2251e712dfc7caa34dcbd297d00dec6b23381762077159b56f5b7a1b1651ecfe"}
Oct 30 22:53:45.189476 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:45.189383    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:53:45.189646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:45.189571    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 22:53:45.189646 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:45.189615    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k" podUID="e86e2368-d03b-4634-8245-eddf9d83e16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 22:53:46.189569 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:46.189530    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 22:53:46.190038 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:46.189584    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k" podUID="e86e2368-d03b-4634-8245-eddf9d83e16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 22:53:46.642277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:46.642078    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:47.189460 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:47.189357    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 22:53:47.189460 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:47.189444    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k" podUID="e86e2368-d03b-4634-8245-eddf9d83e16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 22:53:48.189649 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:48.189606    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 22:53:48.190191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:48.189693    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k" podUID="e86e2368-d03b-4634-8245-eddf9d83e16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 22:53:48.550249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:48.547178    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:49.190056 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:49.190014    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 22:53:49.190543 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:49.190071    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k" podUID="e86e2368-d03b-4634-8245-eddf9d83e16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 22:53:50.190148 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:50.190100    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 22:53:50.190604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:50.190165    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k" podUID="e86e2368-d03b-4634-8245-eddf9d83e16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 22:53:50.337927 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:50.337893    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:50.507292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:50.507218    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:51.195784 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:51.195752    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:53:51.196282 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:51.196262    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:53:51.197110 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:51.197089    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 22:53:52.130122 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:52.130085    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:53.830548 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:53.830514    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:55.626869 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:55.626833    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:57.417457 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:57.417412    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:59.210283 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:53:59.210200    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:00.906298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:00.903876    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:02.653349 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:02.653309    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:04.527155 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:04.527113    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:06.358369 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:06.358327    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:08.614783 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:08.614689    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:10.292167 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:10.292103    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:10.319570 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:54:10.319527    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:54:12.044983 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:12.044944    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:13.831900 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:13.831858    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:15.495248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:15.495192    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:17.386072 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:17.386038    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:19.126299 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:19.126207    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:20.846393 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:20.846344    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:21.550726 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:21.550693    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m"]
Oct 30 22:54:21.585817 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:21.585766    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68"]
Oct 30 22:54:21.752772 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:21.752731    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:21.931355 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:21.931257    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:22.632869 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:22.632819    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:24.339925 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:24.339881    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:26.193834 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:26.193772    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:26.551994 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:26.551950    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m" podUID="53046f1e-ac05-4261-8258-b6109ecdea33" containerName="payload-processing" containerID="cri-o://84f634ce947a3652a4dddd9e7733ee5ccc91493a08b5ec82c57f1b04bfc3c0a4" gracePeriod=25
Oct 30 22:54:26.586255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:26.586180    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68" podUID="b2006c00-f096-4380-b39e-09c52d3f7ee0" containerName="payload-pre-processing" containerID="cri-o://a0b4e53ca0cc2e7cf8e70759536bed5fcdc5a985893609bb1f827732f12a751d" gracePeriod=25
Oct 30 22:54:26.900878 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:26.900846    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m"
Oct 30 22:54:26.939626 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:26.939599    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68"
Oct 30 22:54:27.021946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.021904    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53046f1e-ac05-4261-8258-b6109ecdea33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53046f1e-ac05-4261-8258-b6109ecdea33-plugins-config-volume\") pod \"53046f1e-ac05-4261-8258-b6109ecdea33\" (UID: \"53046f1e-ac05-4261-8258-b6109ecdea33\") "
Oct 30 22:54:27.022158 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.022086    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53046f1e-ac05-4261-8258-b6109ecdea33-kube-api-access-99xbn\" (UniqueName: \"kubernetes.io/projected/53046f1e-ac05-4261-8258-b6109ecdea33-kube-api-access-99xbn\") pod \"53046f1e-ac05-4261-8258-b6109ecdea33\" (UID: \"53046f1e-ac05-4261-8258-b6109ecdea33\") "
Oct 30 22:54:27.022251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.022175    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2006c00-f096-4380-b39e-09c52d3f7ee0-kube-api-access-fhtzz\" (UniqueName: \"kubernetes.io/projected/b2006c00-f096-4380-b39e-09c52d3f7ee0-kube-api-access-fhtzz\") pod \"b2006c00-f096-4380-b39e-09c52d3f7ee0\" (UID: \"b2006c00-f096-4380-b39e-09c52d3f7ee0\") "
Oct 30 22:54:27.022332 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.022311    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2006c00-f096-4380-b39e-09c52d3f7ee0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2006c00-f096-4380-b39e-09c52d3f7ee0-plugins-config-volume\") pod \"b2006c00-f096-4380-b39e-09c52d3f7ee0\" (UID: \"b2006c00-f096-4380-b39e-09c52d3f7ee0\") "
Oct 30 22:54:27.023640 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.023608    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53046f1e-ac05-4261-8258-b6109ecdea33-plugins-config-volume" pod "53046f1e-ac05-4261-8258-b6109ecdea33" (UID: "53046f1e-ac05-4261-8258-b6109ecdea33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:27.023808 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.023585    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2006c00-f096-4380-b39e-09c52d3f7ee0-plugins-config-volume" pod "b2006c00-f096-4380-b39e-09c52d3f7ee0" (UID: "b2006c00-f096-4380-b39e-09c52d3f7ee0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:27.026460 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.026430    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2006c00-f096-4380-b39e-09c52d3f7ee0-kube-api-access-fhtzz" pod "b2006c00-f096-4380-b39e-09c52d3f7ee0" (UID: "b2006c00-f096-4380-b39e-09c52d3f7ee0"). InnerVolumeSpecName "kube-api-access-fhtzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:27.026684 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.026657    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53046f1e-ac05-4261-8258-b6109ecdea33-kube-api-access-99xbn" pod "53046f1e-ac05-4261-8258-b6109ecdea33" (UID: "53046f1e-ac05-4261-8258-b6109ecdea33"). InnerVolumeSpecName "kube-api-access-99xbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:27.124422 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.124315    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99xbn\" (UniqueName: \"kubernetes.io/projected/53046f1e-ac05-4261-8258-b6109ecdea33-kube-api-access-99xbn\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:54:27.124422 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.124361    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhtzz\" (UniqueName: \"kubernetes.io/projected/b2006c00-f096-4380-b39e-09c52d3f7ee0-kube-api-access-fhtzz\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:54:27.124422 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.124376    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2006c00-f096-4380-b39e-09c52d3f7ee0-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:54:27.124422 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.124389    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53046f1e-ac05-4261-8258-b6109ecdea33-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:54:27.262687 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.262645    4889 generic.go:356] "Generic (PLEG): container finished" podID="53046f1e-ac05-4261-8258-b6109ecdea33" containerID="84f634ce947a3652a4dddd9e7733ee5ccc91493a08b5ec82c57f1b04bfc3c0a4" exitCode=0
Oct 30 22:54:27.263328 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.262699    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m"
Oct 30 22:54:27.263328 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.262729    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m" event={"ID":"53046f1e-ac05-4261-8258-b6109ecdea33","Type":"ContainerDied","Data":"84f634ce947a3652a4dddd9e7733ee5ccc91493a08b5ec82c57f1b04bfc3c0a4"}
Oct 30 22:54:27.263328 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.262786    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m" event={"ID":"53046f1e-ac05-4261-8258-b6109ecdea33","Type":"ContainerDied","Data":"163ca8219cf78036f8734444c5a3f3b3429ebe7b10b02a91896ed1535c55868b"}
Oct 30 22:54:27.263328 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.262809    4889 scope.go:122] "RemoveContainer" containerID="84f634ce947a3652a4dddd9e7733ee5ccc91493a08b5ec82c57f1b04bfc3c0a4"
Oct 30 22:54:27.264730 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.264698    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68" event={"ID":"b2006c00-f096-4380-b39e-09c52d3f7ee0","Type":"ContainerDied","Data":"a0b4e53ca0cc2e7cf8e70759536bed5fcdc5a985893609bb1f827732f12a751d"}
Oct 30 22:54:27.264730 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.264713    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68"
Oct 30 22:54:27.264899 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.264736    4889 generic.go:356] "Generic (PLEG): container finished" podID="b2006c00-f096-4380-b39e-09c52d3f7ee0" containerID="a0b4e53ca0cc2e7cf8e70759536bed5fcdc5a985893609bb1f827732f12a751d" exitCode=0
Oct 30 22:54:27.264899 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.264761    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68" event={"ID":"b2006c00-f096-4380-b39e-09c52d3f7ee0","Type":"ContainerDied","Data":"1d1bdfd8f3276520451ee1bf3f1dfdf5c5d6e00f0c04678639ecbe7c8d138cc7"}
Oct 30 22:54:27.275778 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.275750    4889 scope.go:122] "RemoveContainer" containerID="84f634ce947a3652a4dddd9e7733ee5ccc91493a08b5ec82c57f1b04bfc3c0a4"
Oct 30 22:54:27.276158 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:54:27.276126    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84f634ce947a3652a4dddd9e7733ee5ccc91493a08b5ec82c57f1b04bfc3c0a4\": container with ID starting with 84f634ce947a3652a4dddd9e7733ee5ccc91493a08b5ec82c57f1b04bfc3c0a4 not found: ID does not exist" containerID="84f634ce947a3652a4dddd9e7733ee5ccc91493a08b5ec82c57f1b04bfc3c0a4"
Oct 30 22:54:27.276321 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.276168    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84f634ce947a3652a4dddd9e7733ee5ccc91493a08b5ec82c57f1b04bfc3c0a4"} err="failed to get container status \"84f634ce947a3652a4dddd9e7733ee5ccc91493a08b5ec82c57f1b04bfc3c0a4\": rpc error: code = NotFound desc = could not find container \"84f634ce947a3652a4dddd9e7733ee5ccc91493a08b5ec82c57f1b04bfc3c0a4\": container with ID starting with 84f634ce947a3652a4dddd9e7733ee5ccc91493a08b5ec82c57f1b04bfc3c0a4 not found: ID does not exist"
Oct 30 22:54:27.276321 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.276193    4889 scope.go:122] "RemoveContainer" containerID="a0b4e53ca0cc2e7cf8e70759536bed5fcdc5a985893609bb1f827732f12a751d"
Oct 30 22:54:27.289746 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.289717    4889 scope.go:122] "RemoveContainer" containerID="a0b4e53ca0cc2e7cf8e70759536bed5fcdc5a985893609bb1f827732f12a751d"
Oct 30 22:54:27.290117 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:54:27.290098    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0b4e53ca0cc2e7cf8e70759536bed5fcdc5a985893609bb1f827732f12a751d\": container with ID starting with a0b4e53ca0cc2e7cf8e70759536bed5fcdc5a985893609bb1f827732f12a751d not found: ID does not exist" containerID="a0b4e53ca0cc2e7cf8e70759536bed5fcdc5a985893609bb1f827732f12a751d"
Oct 30 22:54:27.290201 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.290123    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0b4e53ca0cc2e7cf8e70759536bed5fcdc5a985893609bb1f827732f12a751d"} err="failed to get container status \"a0b4e53ca0cc2e7cf8e70759536bed5fcdc5a985893609bb1f827732f12a751d\": rpc error: code = NotFound desc = could not find container \"a0b4e53ca0cc2e7cf8e70759536bed5fcdc5a985893609bb1f827732f12a751d\": container with ID starting with a0b4e53ca0cc2e7cf8e70759536bed5fcdc5a985893609bb1f827732f12a751d not found: ID does not exist"
Oct 30 22:54:27.321734 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.321685    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m"]
Oct 30 22:54:27.325929 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.325894    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a60e80-7b8469d759-5hv8m"]
Oct 30 22:54:27.364920 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.364877    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68"]
Oct 30 22:54:27.375574 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:27.375479    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5949b5b66c-h6x68"]
Oct 30 22:54:28.012409 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:28.012364    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:28.777442 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:28.777406    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53046f1e-ac05-4261-8258-b6109ecdea33" path="/var/lib/kubelet/pods/53046f1e-ac05-4261-8258-b6109ecdea33/volumes"
Oct 30 22:54:28.778133 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:28.778113    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2006c00-f096-4380-b39e-09c52d3f7ee0" path="/var/lib/kubelet/pods/b2006c00-f096-4380-b39e-09c52d3f7ee0/volumes"
Oct 30 22:54:29.847864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:29.847826    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:31.790983 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:31.790942    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:33.551367 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:33.551329    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:35.339631 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:35.339593    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:37.055204 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:37.055164    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:37.223983 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:37.223949    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:37.763926 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:37.763876    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:38.830643 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:38.830571    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:40.305005 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:54:40.304967    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:54:40.590939 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:40.590836    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:42.395162 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:42.395127    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:44.183471 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:44.183424    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:45.955674 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:45.955634    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:47.705986 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:47.705937    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:49.645521 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:49.645482    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:50.589913 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:50.589871    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:51.880587 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:51.880544    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:53.678874 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:53.678836    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:55.442477 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:55.442431    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:57.181287 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:57.181205    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:59.108061 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:54:59.108019    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:00.356686 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:00.356640    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:00.534862 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:00.534824    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:01.626790 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:01.623806    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:03.317285 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:03.317250    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:05.088353 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:05.088310    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:06.818961 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:06.818905    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:08.588394 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:08.588356    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:10.308189 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:55:10.308150    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:55:10.318326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:10.318289    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:12.125860 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:12.125819    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:12.643330 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:12.643291    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:13.919685 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:13.919581    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:15.652941 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:15.652903    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:17.456648 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:17.456601    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:19.234623 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:19.234586    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:20.927858 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:20.927819    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:22.863782 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:22.863709    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:23.114497 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:23.114393    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rp6t6_f32ffa1e-863c-4df9-b4d1-f80463ed099c/payload-processing/0.log"
Oct 30 22:55:23.309716 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:23.309681    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rp6t6_f32ffa1e-863c-4df9-b4d1-f80463ed099c/payload-processing/0.log"
Oct 30 22:55:24.815111 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:24.815073    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:26.809300 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:26.809197    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:27.478889 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:27.478851    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:28.556838 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:28.556795    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:30.261042 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:30.261001    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:31.951046 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:31.951004    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:33.646199 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:33.646156    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:35.891979 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:35.891873    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:37.616872 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:37.616830    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:39.344842 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:39.344808    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:40.304675 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:55:40.304634    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:55:41.163911 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:41.163873    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:43.072719 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:43.072681    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:45.336864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:45.336816    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:47.214590 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:47.214478    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:49.153636 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:49.153604    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:50.737956 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:50.737917    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:50.916329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:50.916286    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:52.677428 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:52.677388    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:54.519612 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:54.519554    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:56.343577 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:56.343537    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:58.137757 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:58.137720    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:59.903439 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:55:59.903405    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:01.581636 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:01.581593    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:03.330819 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:03.330784    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:05.022040 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:05.022004    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:05.193788 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:05.193732    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:05.456681 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:05.456567    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"]
Oct 30 22:56:05.622829 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:05.622793    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:05.937146 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:05.937102    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"]
Oct 30 22:56:06.046428 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:06.046384    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:06.858106 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:06.858068    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:07.046835 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:07.046797    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:08.390157 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:08.390119    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"]
Oct 30 22:56:08.472084 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:08.472049    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:08.650296 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:08.650190    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:09.107482 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:09.107443    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:09.594538 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:09.594494    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:09.769001 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:09.768966    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:10.304408 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:56:10.304368    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:56:10.439492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:10.439442    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:12.272486 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:12.272412    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:13.273185 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:13.273150    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:13.463840 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:13.463776    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:14.190170 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:14.190138    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:14.928573 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:14.928533    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:15.143281 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:15.129683    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:15.155634 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:15.155579    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8"]
Oct 30 22:56:15.157397 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:15.157196    4889 cpu_manager.go:414] "containerMap: removing container" podUID="b2006c00-f096-4380-b39e-09c52d3f7ee0" containerName="payload-pre-processing"
Oct 30 22:56:15.164249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:15.158977    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2006c00-f096-4380-b39e-09c52d3f7ee0" containerName="payload-pre-processing"
Oct 30 22:56:15.164608 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:15.164584    4889 cpu_manager.go:414] "containerMap: removing container" podUID="53046f1e-ac05-4261-8258-b6109ecdea33" containerName="payload-processing"
Oct 30 22:56:15.164763 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:15.164747    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53046f1e-ac05-4261-8258-b6109ecdea33" containerName="payload-processing"
Oct 30 22:56:15.165177 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:15.165157    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2006c00-f096-4380-b39e-09c52d3f7ee0" containerName="payload-pre-processing"
Oct 30 22:56:15.165334 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:15.165320    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="53046f1e-ac05-4261-8258-b6109ecdea33" containerName="payload-processing"
Oct 30 22:56:15.176506 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:15.175200    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8"
Oct 30 22:56:15.182663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:15.182558    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8"]
Oct 30 22:56:15.195259 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:15.193604    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hm6n\" (UniqueName: \"kubernetes.io/projected/512c5c51-57a4-48d3-b1c3-e1fdb7bbd483-kube-api-access-2hm6n\") pod \"payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8\" (UID: \"512c5c51-57a4-48d3-b1c3-e1fdb7bbd483\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8"
Oct 30 22:56:15.195259 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:15.193700    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/512c5c51-57a4-48d3-b1c3-e1fdb7bbd483-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8\" (UID: \"512c5c51-57a4-48d3-b1c3-e1fdb7bbd483\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8"
Oct 30 22:56:15.208686 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:15.208658    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f0a807\""
Oct 30 22:56:15.209081 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:15.209065    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f0a807-dockercfg-rxskz\""
Oct 30 22:56:15.295053 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:15.295007    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hm6n\" (UniqueName: \"kubernetes.io/projected/512c5c51-57a4-48d3-b1c3-e1fdb7bbd483-kube-api-access-2hm6n\") pod \"payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8\" (UID: \"512c5c51-57a4-48d3-b1c3-e1fdb7bbd483\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8"
Oct 30 22:56:15.295326 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:15.295094    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/512c5c51-57a4-48d3-b1c3-e1fdb7bbd483-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8\" (UID: \"512c5c51-57a4-48d3-b1c3-e1fdb7bbd483\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8"
Oct 30 22:56:15.295888 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:15.295862    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/512c5c51-57a4-48d3-b1c3-e1fdb7bbd483-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8\" (UID: \"512c5c51-57a4-48d3-b1c3-e1fdb7bbd483\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8"
Oct 30 22:56:15.316308 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:15.316271    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hm6n\" (UniqueName: \"kubernetes.io/projected/512c5c51-57a4-48d3-b1c3-e1fdb7bbd483-kube-api-access-2hm6n\") pod \"payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8\" (UID: \"512c5c51-57a4-48d3-b1c3-e1fdb7bbd483\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8"
Oct 30 22:56:15.519820 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:15.519786    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8"
Oct 30 22:56:15.834753 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:56:15.834720    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512c5c51_57a4_48d3_b1c3_e1fdb7bbd483.slice/crio-61630b4b627b6ab4dea053cba3cd892f0099720e8f2a09fb99b8cafc8c13e921 WatchSource:0}: Error finding container 61630b4b627b6ab4dea053cba3cd892f0099720e8f2a09fb99b8cafc8c13e921: Status 404 returned error can't find the container with id 61630b4b627b6ab4dea053cba3cd892f0099720e8f2a09fb99b8cafc8c13e921
Oct 30 22:56:15.835082 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:15.835054    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8"]
Oct 30 22:56:15.937594 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:15.937555    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:16.288178 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:16.288142    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8" event={"ID":"512c5c51-57a4-48d3-b1c3-e1fdb7bbd483","Type":"ContainerStarted","Data":"0a661806e131496ac6322cc2a6413738de2e0aec3c285c1bb83eacd28ce0bfe5"}
Oct 30 22:56:16.288444 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:16.288204    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8" event={"ID":"512c5c51-57a4-48d3-b1c3-e1fdb7bbd483","Type":"ContainerStarted","Data":"61630b4b627b6ab4dea053cba3cd892f0099720e8f2a09fb99b8cafc8c13e921"}
Oct 30 22:56:16.288444 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:16.288390    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8"
Oct 30 22:56:16.331535 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:16.331468    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8" podStartSLOduration=1.331452975 podStartE2EDuration="1.331452975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:56:16.330245913 +0000 UTC m=+2961.686612225" watchObservedRunningTime="2026-10-30 22:56:16.331452975 +0000 UTC m=+2961.687819291"
Oct 30 22:56:17.206073 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:17.206025    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:17.552287 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:17.549199    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"]
Oct 30 22:56:17.640104 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:17.640069    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:17.990598 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:17.990562    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:20.005606 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:20.005566    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:20.481996 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:20.481907    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj" podUID="fe7f6368-5fa6-4f71-8dab-9283198e9c13" containerName="main" containerID="cri-o://8aac8fc244bee4779d34f2cc88d7f9b01148959ca17c7c3a4a912617845d5995" gracePeriod=45
Oct 30 22:56:20.566515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:20.566479    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:20.799693 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:20.799663    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"
Oct 30 22:56:20.903506 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:20.903463    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe7f6368-5fa6-4f71-8dab-9283198e9c13-kube-api-access-gg4ct\" (UniqueName: \"kubernetes.io/projected/fe7f6368-5fa6-4f71-8dab-9283198e9c13-kube-api-access-gg4ct\") pod \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\" (UID: \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\") "
Oct 30 22:56:20.903730 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:20.903614    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-model-cache\") pod \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\" (UID: \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\") "
Oct 30 22:56:20.903730 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:20.903708    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-dshm\") pod \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\" (UID: \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\") "
Oct 30 22:56:20.903903 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:20.903862    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-tmp-dir\") pod \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\" (UID: \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\") "
Oct 30 22:56:20.903903 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:20.903894    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-home\" (UniqueName: \"kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-home\") pod \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\" (UID: \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\") "
Oct 30 22:56:20.904010 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:20.903940    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-model-cache" pod "fe7f6368-5fa6-4f71-8dab-9283198e9c13" (UID: "fe7f6368-5fa6-4f71-8dab-9283198e9c13"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:20.904010 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:20.904001    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe7f6368-5fa6-4f71-8dab-9283198e9c13-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe7f6368-5fa6-4f71-8dab-9283198e9c13-tls-certs\") pod \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\" (UID: \"fe7f6368-5fa6-4f71-8dab-9283198e9c13\") "
Oct 30 22:56:20.904236 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:20.904127    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-tmp-dir" pod "fe7f6368-5fa6-4f71-8dab-9283198e9c13" (UID: "fe7f6368-5fa6-4f71-8dab-9283198e9c13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:20.904236 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:20.904169    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-home" pod "fe7f6368-5fa6-4f71-8dab-9283198e9c13" (UID: "fe7f6368-5fa6-4f71-8dab-9283198e9c13"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:20.905017 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:20.904991    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:20.905169 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:20.905026    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:20.905169 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:20.905042    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:20.906674 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:20.906647    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe7f6368-5fa6-4f71-8dab-9283198e9c13-kube-api-access-gg4ct" pod "fe7f6368-5fa6-4f71-8dab-9283198e9c13" (UID: "fe7f6368-5fa6-4f71-8dab-9283198e9c13"). InnerVolumeSpecName "kube-api-access-gg4ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:20.906793 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:20.906740    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-dshm" pod "fe7f6368-5fa6-4f71-8dab-9283198e9c13" (UID: "fe7f6368-5fa6-4f71-8dab-9283198e9c13"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:20.906877 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:20.906854    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe7f6368-5fa6-4f71-8dab-9283198e9c13-tls-certs" pod "fe7f6368-5fa6-4f71-8dab-9283198e9c13" (UID: "fe7f6368-5fa6-4f71-8dab-9283198e9c13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:20.970217 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:20.970148    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj" podUID="fa03e689-d506-45ca-9883-6627c13d25da" containerName="main" containerID="cri-o://085638291a60cfb021f15c2f0354c6941029dc83a3e8eb7b0564854167490d95" gracePeriod=45
Oct 30 22:56:21.006367 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.006320    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe7f6368-5fa6-4f71-8dab-9283198e9c13-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:21.006367 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.006351    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe7f6368-5fa6-4f71-8dab-9283198e9c13-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:21.006367 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.006366    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg4ct\" (UniqueName: \"kubernetes.io/projected/fe7f6368-5fa6-4f71-8dab-9283198e9c13-kube-api-access-gg4ct\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:21.119353 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.119306    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj" podUID="fa03e689-d506-45ca-9883-6627c13d25da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.210:8000/ready\": dial tcp 10.132.0.210:8000: connect: connection refused"
Oct 30 22:56:21.274105 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.274076    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"
Oct 30 22:56:21.340976 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.340932    4889 generic.go:356] "Generic (PLEG): container finished" podID="fa03e689-d506-45ca-9883-6627c13d25da" containerID="085638291a60cfb021f15c2f0354c6941029dc83a3e8eb7b0564854167490d95" exitCode=0
Oct 30 22:56:21.341193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.341056    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"
Oct 30 22:56:21.341193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.341139    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj" event={"ID":"fa03e689-d506-45ca-9883-6627c13d25da","Type":"ContainerDied","Data":"085638291a60cfb021f15c2f0354c6941029dc83a3e8eb7b0564854167490d95"}
Oct 30 22:56:21.341193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.341179    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj" event={"ID":"fa03e689-d506-45ca-9883-6627c13d25da","Type":"ContainerDied","Data":"5e0ec6ba8453767dbd11b313264b224cadda7eed2ddd4b9d08c0efcb34021bbb"}
Oct 30 22:56:21.341513 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.341201    4889 scope.go:122] "RemoveContainer" containerID="085638291a60cfb021f15c2f0354c6941029dc83a3e8eb7b0564854167490d95"
Oct 30 22:56:21.344578 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.344503    4889 generic.go:356] "Generic (PLEG): container finished" podID="fe7f6368-5fa6-4f71-8dab-9283198e9c13" containerID="8aac8fc244bee4779d34f2cc88d7f9b01148959ca17c7c3a4a912617845d5995" exitCode=0
Oct 30 22:56:21.344578 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.344543    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"
Oct 30 22:56:21.344578 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.344566    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj" event={"ID":"fe7f6368-5fa6-4f71-8dab-9283198e9c13","Type":"ContainerDied","Data":"8aac8fc244bee4779d34f2cc88d7f9b01148959ca17c7c3a4a912617845d5995"}
Oct 30 22:56:21.344910 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.344615    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj" event={"ID":"fe7f6368-5fa6-4f71-8dab-9283198e9c13","Type":"ContainerDied","Data":"386fd50543c06d275d3fa256e02ec1d04fad0c309b3cf2861831e20cad549892"}
Oct 30 22:56:21.358070 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.357150    4889 scope.go:122] "RemoveContainer" containerID="085638291a60cfb021f15c2f0354c6941029dc83a3e8eb7b0564854167490d95"
Oct 30 22:56:21.358201 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:56:21.358120    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"085638291a60cfb021f15c2f0354c6941029dc83a3e8eb7b0564854167490d95\": container with ID starting with 085638291a60cfb021f15c2f0354c6941029dc83a3e8eb7b0564854167490d95 not found: ID does not exist" containerID="085638291a60cfb021f15c2f0354c6941029dc83a3e8eb7b0564854167490d95"
Oct 30 22:56:21.358201 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.358160    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"085638291a60cfb021f15c2f0354c6941029dc83a3e8eb7b0564854167490d95"} err="failed to get container status \"085638291a60cfb021f15c2f0354c6941029dc83a3e8eb7b0564854167490d95\": rpc error: code = NotFound desc = could not find container \"085638291a60cfb021f15c2f0354c6941029dc83a3e8eb7b0564854167490d95\": container with ID starting with 085638291a60cfb021f15c2f0354c6941029dc83a3e8eb7b0564854167490d95 not found: ID does not exist"
Oct 30 22:56:21.358201 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.358186    4889 scope.go:122] "RemoveContainer" containerID="8aac8fc244bee4779d34f2cc88d7f9b01148959ca17c7c3a4a912617845d5995"
Oct 30 22:56:21.371289 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.371265    4889 scope.go:122] "RemoveContainer" containerID="8aac8fc244bee4779d34f2cc88d7f9b01148959ca17c7c3a4a912617845d5995"
Oct 30 22:56:21.371660 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:56:21.371637    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aac8fc244bee4779d34f2cc88d7f9b01148959ca17c7c3a4a912617845d5995\": container with ID starting with 8aac8fc244bee4779d34f2cc88d7f9b01148959ca17c7c3a4a912617845d5995 not found: ID does not exist" containerID="8aac8fc244bee4779d34f2cc88d7f9b01148959ca17c7c3a4a912617845d5995"
Oct 30 22:56:21.371731 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.371672    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aac8fc244bee4779d34f2cc88d7f9b01148959ca17c7c3a4a912617845d5995"} err="failed to get container status \"8aac8fc244bee4779d34f2cc88d7f9b01148959ca17c7c3a4a912617845d5995\": rpc error: code = NotFound desc = could not find container \"8aac8fc244bee4779d34f2cc88d7f9b01148959ca17c7c3a4a912617845d5995\": container with ID starting with 8aac8fc244bee4779d34f2cc88d7f9b01148959ca17c7c3a4a912617845d5995 not found: ID does not exist"
Oct 30 22:56:21.401545 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.401502    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"]
Oct 30 22:56:21.405705 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.405667    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c511a3/auth-test-model-c511a3-kserve-6656df857c-hlbzj"]
Oct 30 22:56:21.410885 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.410854    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa03e689-d506-45ca-9883-6627c13d25da-kube-api-access-zjkgq\" (UniqueName: \"kubernetes.io/projected/fa03e689-d506-45ca-9883-6627c13d25da-kube-api-access-zjkgq\") pod \"fa03e689-d506-45ca-9883-6627c13d25da\" (UID: \"fa03e689-d506-45ca-9883-6627c13d25da\") "
Oct 30 22:56:21.411050 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.410896    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa03e689-d506-45ca-9883-6627c13d25da-tls-certs\" (UniqueName: \"kubernetes.io/secret/fa03e689-d506-45ca-9883-6627c13d25da-tls-certs\") pod \"fa03e689-d506-45ca-9883-6627c13d25da\" (UID: \"fa03e689-d506-45ca-9883-6627c13d25da\") "
Oct 30 22:56:21.411106 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.411058    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-tmp-dir\") pod \"fa03e689-d506-45ca-9883-6627c13d25da\" (UID: \"fa03e689-d506-45ca-9883-6627c13d25da\") "
Oct 30 22:56:21.411163 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.411135    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-model-cache\") pod \"fa03e689-d506-45ca-9883-6627c13d25da\" (UID: \"fa03e689-d506-45ca-9883-6627c13d25da\") "
Oct 30 22:56:21.411320 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.411290    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-dshm\") pod \"fa03e689-d506-45ca-9883-6627c13d25da\" (UID: \"fa03e689-d506-45ca-9883-6627c13d25da\") "
Oct 30 22:56:21.411409 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.411337    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-home\" (UniqueName: \"kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-home\") pod \"fa03e689-d506-45ca-9883-6627c13d25da\" (UID: \"fa03e689-d506-45ca-9883-6627c13d25da\") "
Oct 30 22:56:21.411409 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.411385    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-tmp-dir" pod "fa03e689-d506-45ca-9883-6627c13d25da" (UID: "fa03e689-d506-45ca-9883-6627c13d25da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:21.411512 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.411429    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-model-cache" pod "fa03e689-d506-45ca-9883-6627c13d25da" (UID: "fa03e689-d506-45ca-9883-6627c13d25da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:21.411777 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.411676    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-home" pod "fa03e689-d506-45ca-9883-6627c13d25da" (UID: "fa03e689-d506-45ca-9883-6627c13d25da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:21.412334 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.412313    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:21.412419 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.412339    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:21.412419 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.412356    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:21.413692 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.413666    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-dshm" pod "fa03e689-d506-45ca-9883-6627c13d25da" (UID: "fa03e689-d506-45ca-9883-6627c13d25da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:21.413692 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.413677    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa03e689-d506-45ca-9883-6627c13d25da-tls-certs" pod "fa03e689-d506-45ca-9883-6627c13d25da" (UID: "fa03e689-d506-45ca-9883-6627c13d25da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:21.413830 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.413722    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa03e689-d506-45ca-9883-6627c13d25da-kube-api-access-zjkgq" pod "fa03e689-d506-45ca-9883-6627c13d25da" (UID: "fa03e689-d506-45ca-9883-6627c13d25da"). InnerVolumeSpecName "kube-api-access-zjkgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:21.514284 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.514207    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa03e689-d506-45ca-9883-6627c13d25da-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:21.514444 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.514290    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjkgq\" (UniqueName: \"kubernetes.io/projected/fa03e689-d506-45ca-9883-6627c13d25da-kube-api-access-zjkgq\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:21.514444 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.514306    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa03e689-d506-45ca-9883-6627c13d25da-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:21.692903 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.692866    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"]
Oct 30 22:56:21.696864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:21.696832    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-6f2eba/auth-test-model-6f2eba-kserve-66d59bcb9c-m8wpj"]
Oct 30 22:56:22.442446 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:22.442409    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:22.777522 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:22.777485    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa03e689-d506-45ca-9883-6627c13d25da" path="/var/lib/kubelet/pods/fa03e689-d506-45ca-9883-6627c13d25da/volumes"
Oct 30 22:56:22.778193 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:22.778172    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe7f6368-5fa6-4f71-8dab-9283198e9c13" path="/var/lib/kubelet/pods/fe7f6368-5fa6-4f71-8dab-9283198e9c13/volumes"
Oct 30 22:56:23.414429 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:23.414387    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx" podUID="f0a827e1-5ee3-49bf-af24-c538d0c211ef" containerName="main" containerID="cri-o://a04684de8e2a6aecde14261eecdc5ab9f895de0e85437fa93a173029bfb8aab9" gracePeriod=45
Oct 30 22:56:23.721421 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:23.721391    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"
Oct 30 22:56:23.853362 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:23.853317    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-dshm\") pod \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\" (UID: \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\") "
Oct 30 22:56:23.853557 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:23.853438    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-model-cache\") pod \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\" (UID: \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\") "
Oct 30 22:56:23.853557 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:23.853505    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0a827e1-5ee3-49bf-af24-c538d0c211ef-kube-api-access-vlvbh\" (UniqueName: \"kubernetes.io/projected/f0a827e1-5ee3-49bf-af24-c538d0c211ef-kube-api-access-vlvbh\") pod \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\" (UID: \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\") "
Oct 30 22:56:23.853762 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:23.853567    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0a827e1-5ee3-49bf-af24-c538d0c211ef-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0a827e1-5ee3-49bf-af24-c538d0c211ef-tls-certs\") pod \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\" (UID: \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\") "
Oct 30 22:56:23.853762 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:23.853613    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-tmp-dir\") pod \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\" (UID: \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\") "
Oct 30 22:56:23.853762 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:23.853668    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-home\" (UniqueName: \"kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-home\") pod \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\" (UID: \"f0a827e1-5ee3-49bf-af24-c538d0c211ef\") "
Oct 30 22:56:23.853762 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:23.853727    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-model-cache" pod "f0a827e1-5ee3-49bf-af24-c538d0c211ef" (UID: "f0a827e1-5ee3-49bf-af24-c538d0c211ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:23.853991 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:23.853874    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-tmp-dir" pod "f0a827e1-5ee3-49bf-af24-c538d0c211ef" (UID: "f0a827e1-5ee3-49bf-af24-c538d0c211ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:23.853991 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:23.853971    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-home" pod "f0a827e1-5ee3-49bf-af24-c538d0c211ef" (UID: "f0a827e1-5ee3-49bf-af24-c538d0c211ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:23.854788 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:23.854764    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:23.854889 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:23.854794    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:23.854889 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:23.854810    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:23.856045 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:23.856018    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-dshm" pod "f0a827e1-5ee3-49bf-af24-c538d0c211ef" (UID: "f0a827e1-5ee3-49bf-af24-c538d0c211ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:23.856161 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:23.856089    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0a827e1-5ee3-49bf-af24-c538d0c211ef-tls-certs" pod "f0a827e1-5ee3-49bf-af24-c538d0c211ef" (UID: "f0a827e1-5ee3-49bf-af24-c538d0c211ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:23.856314 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:23.856294    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a827e1-5ee3-49bf-af24-c538d0c211ef-kube-api-access-vlvbh" pod "f0a827e1-5ee3-49bf-af24-c538d0c211ef" (UID: "f0a827e1-5ee3-49bf-af24-c538d0c211ef"). InnerVolumeSpecName "kube-api-access-vlvbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:23.956120 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:23.956031    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0a827e1-5ee3-49bf-af24-c538d0c211ef-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:23.956120 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:23.956063    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0a827e1-5ee3-49bf-af24-c538d0c211ef-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:23.956120 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:23.956072    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlvbh\" (UniqueName: \"kubernetes.io/projected/f0a827e1-5ee3-49bf-af24-c538d0c211ef-kube-api-access-vlvbh\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:24.143297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:24.143204    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:24.372578 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:24.372543    4889 generic.go:356] "Generic (PLEG): container finished" podID="f0a827e1-5ee3-49bf-af24-c538d0c211ef" containerID="a04684de8e2a6aecde14261eecdc5ab9f895de0e85437fa93a173029bfb8aab9" exitCode=0
Oct 30 22:56:24.372784 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:24.372585    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"
Oct 30 22:56:24.372784 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:24.372626    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx" event={"ID":"f0a827e1-5ee3-49bf-af24-c538d0c211ef","Type":"ContainerDied","Data":"a04684de8e2a6aecde14261eecdc5ab9f895de0e85437fa93a173029bfb8aab9"}
Oct 30 22:56:24.372784 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:24.372672    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx" event={"ID":"f0a827e1-5ee3-49bf-af24-c538d0c211ef","Type":"ContainerDied","Data":"3c80346bdb648e28e03a7d2a94101f425daa29eecbd0a071b44ade53d865dc20"}
Oct 30 22:56:24.372784 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:24.372691    4889 scope.go:122] "RemoveContainer" containerID="a04684de8e2a6aecde14261eecdc5ab9f895de0e85437fa93a173029bfb8aab9"
Oct 30 22:56:24.386312 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:24.386289    4889 scope.go:122] "RemoveContainer" containerID="a04684de8e2a6aecde14261eecdc5ab9f895de0e85437fa93a173029bfb8aab9"
Oct 30 22:56:24.386677 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:56:24.386653    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a04684de8e2a6aecde14261eecdc5ab9f895de0e85437fa93a173029bfb8aab9\": container with ID starting with a04684de8e2a6aecde14261eecdc5ab9f895de0e85437fa93a173029bfb8aab9 not found: ID does not exist" containerID="a04684de8e2a6aecde14261eecdc5ab9f895de0e85437fa93a173029bfb8aab9"
Oct 30 22:56:24.386733 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:24.386690    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a04684de8e2a6aecde14261eecdc5ab9f895de0e85437fa93a173029bfb8aab9"} err="failed to get container status \"a04684de8e2a6aecde14261eecdc5ab9f895de0e85437fa93a173029bfb8aab9\": rpc error: code = NotFound desc = could not find container \"a04684de8e2a6aecde14261eecdc5ab9f895de0e85437fa93a173029bfb8aab9\": container with ID starting with a04684de8e2a6aecde14261eecdc5ab9f895de0e85437fa93a173029bfb8aab9 not found: ID does not exist"
Oct 30 22:56:24.431803 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:24.431767    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"]
Oct 30 22:56:24.445631 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:24.445593    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6af86e/auth-test-model-6af86e-kserve-568b8dc989-cdkfx"]
Oct 30 22:56:24.776962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:24.776926    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0a827e1-5ee3-49bf-af24-c538d0c211ef" path="/var/lib/kubelet/pods/f0a827e1-5ee3-49bf-af24-c538d0c211ef/volumes"
Oct 30 22:56:26.049288 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:26.049208    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:27.206074 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:27.198011    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"]
Oct 30 22:56:27.218659 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:27.218624    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:27.305295 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:27.302161    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8"
Oct 30 22:56:27.829604 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:27.829567    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:27.872351 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:27.872318    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg"]
Oct 30 22:56:27.915964 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:27.915924    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2"]
Oct 30 22:56:28.082269 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.082163    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:28.155664 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.155630    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"]
Oct 30 22:56:28.156696 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.156667    4889 cpu_manager.go:414] "containerMap: removing container" podUID="fe7f6368-5fa6-4f71-8dab-9283198e9c13" containerName="main"
Oct 30 22:56:28.156696 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.156696    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe7f6368-5fa6-4f71-8dab-9283198e9c13" containerName="main"
Oct 30 22:56:28.157071 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.156760    4889 cpu_manager.go:414] "containerMap: removing container" podUID="f0a827e1-5ee3-49bf-af24-c538d0c211ef" containerName="main"
Oct 30 22:56:28.157071 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.156768    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0a827e1-5ee3-49bf-af24-c538d0c211ef" containerName="main"
Oct 30 22:56:28.157071 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.156796    4889 cpu_manager.go:414] "containerMap: removing container" podUID="fa03e689-d506-45ca-9883-6627c13d25da" containerName="main"
Oct 30 22:56:28.157071 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.156804    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa03e689-d506-45ca-9883-6627c13d25da" containerName="main"
Oct 30 22:56:28.157071 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.157011    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe7f6368-5fa6-4f71-8dab-9283198e9c13" containerName="main"
Oct 30 22:56:28.157071 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.157042    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0a827e1-5ee3-49bf-af24-c538d0c211ef" containerName="main"
Oct 30 22:56:28.157071 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.157058    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa03e689-d506-45ca-9883-6627c13d25da" containerName="main"
Oct 30 22:56:28.162752 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.162726    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.181421 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.181391    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"]
Oct 30 22:56:28.181662 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.181647    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f88979-openshift-default-dockercfg-7qhlj\""
Oct 30 22:56:28.247859 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.247809    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:28.360082 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.353310    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-istio-data\") pod \"e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") " pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.383376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.383332    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-workload-socket\") pod \"e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") " pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.383686 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.383670    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-istio-envoy\") pod \"e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") " pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.387250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.383856    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2279298-a1a4-469d-a188-f41478b6a598-istiod-ca-cert\") pod \"e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") " pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.414356 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.414310    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-credential-socket\") pod \"e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") " pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.414718 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.414699    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2279298-a1a4-469d-a188-f41478b6a598-istio-token\") pod \"e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") " pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.414874 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.414858    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2279298-a1a4-469d-a188-f41478b6a598-istio-podinfo\") pod \"e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") " pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.415064 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.415049    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vns6f\" (UniqueName: \"kubernetes.io/projected/d2279298-a1a4-469d-a188-f41478b6a598-kube-api-access-vns6f\") pod \"e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") " pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.415188 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.415174    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-workload-certs\") pod \"e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") " pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.455249 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.450838    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:28.517215 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.517172    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2279298-a1a4-469d-a188-f41478b6a598-istio-podinfo\") pod \"e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") " pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.517793 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.517760    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vns6f\" (UniqueName: \"kubernetes.io/projected/d2279298-a1a4-469d-a188-f41478b6a598-kube-api-access-vns6f\") pod \"e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") " pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.518007 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.517986    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-workload-certs\") pod \"e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") " pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.518263 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.518240    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-istio-data\") pod \"e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") " pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.518452 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.518433    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-workload-socket\") pod \"e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") " pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.518626 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.518607    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-istio-envoy\") pod \"e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") " pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.518836 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.518814    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2279298-a1a4-469d-a188-f41478b6a598-istiod-ca-cert\") pod \"e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") " pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.518962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.518946    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-credential-socket\") pod \"e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") " pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.519145 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.519128    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2279298-a1a4-469d-a188-f41478b6a598-istio-token\") pod \"e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") " pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.520630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.520591    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-workload-socket\") pod \"e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") " pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.521458 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.521432    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-workload-certs\") pod \"e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") " pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.521853 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.521833    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-istio-data\") pod \"e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") " pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.522677 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.522654    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2279298-a1a4-469d-a188-f41478b6a598-istiod-ca-cert\") pod \"e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") " pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.523407 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.523385    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-credential-socket\") pod \"e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") " pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.534479 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.534444    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-istio-envoy\") pod \"e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") " pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.574251 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.570799    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2279298-a1a4-469d-a188-f41478b6a598-istio-podinfo\") pod \"e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") " pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.582181 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.582139    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vns6f\" (UniqueName: \"kubernetes.io/projected/d2279298-a1a4-469d-a188-f41478b6a598-kube-api-access-vns6f\") pod \"e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") " pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.595076 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.595040    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2279298-a1a4-469d-a188-f41478b6a598-istio-token\") pod \"e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") " pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:28.658357 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.650973    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:28.792929 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:28.785390    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:29.155886 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:56:29.155846    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2279298_a1a4_469d_a188_f41478b6a598.slice/crio-8f2c8aebc65063ce1fb6a2939518279beb518f54cd6568f9d8f48ced225288d7 WatchSource:0}: Error finding container 8f2c8aebc65063ce1fb6a2939518279beb518f54cd6568f9d8f48ced225288d7: Status 404 returned error can't find the container with id 8f2c8aebc65063ce1fb6a2939518279beb518f54cd6568f9d8f48ced225288d7
Oct 30 22:56:29.156183 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:29.156152    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"]
Oct 30 22:56:29.159069 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:29.159032    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:56:29.159175 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:29.159107    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:56:29.159175 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:29.159140    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:56:29.456631 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:29.456528    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g" event={"ID":"d2279298-a1a4-469d-a188-f41478b6a598","Type":"ContainerStarted","Data":"9289fcccac2f5ebe83a35605eccd354110ae8e9ebd91bab76186f3593754995e"}
Oct 30 22:56:29.456631 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:29.456601    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g" event={"ID":"d2279298-a1a4-469d-a188-f41478b6a598","Type":"ContainerStarted","Data":"8f2c8aebc65063ce1fb6a2939518279beb518f54cd6568f9d8f48ced225288d7"}
Oct 30 22:56:29.506578 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:29.506521    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g" podStartSLOduration=1.506504419 podStartE2EDuration="1.506504419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:56:29.503061557 +0000 UTC m=+2974.859427885" watchObservedRunningTime="2026-10-30 22:56:29.506504419 +0000 UTC m=+2974.862870734"
Oct 30 22:56:29.790270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:29.790194    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:30.538489 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:30.538452    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:30.798112 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:30.798017    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:31.472839 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:31.472805    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:31.474390 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:31.474366    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:56:31.888715 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:31.888675    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:32.075052 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:32.075016    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:32.543798 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:32.543752    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:32.574831 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:32.574786    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km" podUID="818acd09-3afc-4d37-8443-cb3e4e45550b" containerName="main" containerID="cri-o://e94ecd6016f4cf70d4660f75b443ca4cc111c67d5316d0a3b9dd6bc3532d7132" gracePeriod=45
Oct 30 22:56:32.623401 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:32.623358    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km" podUID="818acd09-3afc-4d37-8443-cb3e4e45550b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": dial tcp 10.132.0.213:8000: connect: connection refused"
Oct 30 22:56:32.872746 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:32.872711    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg" podUID="e935aad1-ec9a-45e4-b4d4-31d6b84950a1" containerName="payload-processing" containerID="cri-o://7aa54b2d5342eedd14e9e50f6079d7bc370bda3a857f40f4521a57e0c8c2de0c" gracePeriod=25
Oct 30 22:56:32.878723 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:32.878682    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"
Oct 30 22:56:32.917165 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:32.917123    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2" podUID="7d175091-47d2-43ad-b201-1b7c0d8b82a9" containerName="payload-pre-processing" containerID="cri-o://6afbca25923bc23cabb77349037c6545edb90c4ce30a263ece94aeb7f32ef1ee" gracePeriod=25
Oct 30 22:56:33.011739 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.011697    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-dshm\") pod \"818acd09-3afc-4d37-8443-cb3e4e45550b\" (UID: \"818acd09-3afc-4d37-8443-cb3e4e45550b\") "
Oct 30 22:56:33.011960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.011767    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/818acd09-3afc-4d37-8443-cb3e4e45550b-tls-certs\" (UniqueName: \"kubernetes.io/secret/818acd09-3afc-4d37-8443-cb3e4e45550b-tls-certs\") pod \"818acd09-3afc-4d37-8443-cb3e4e45550b\" (UID: \"818acd09-3afc-4d37-8443-cb3e4e45550b\") "
Oct 30 22:56:33.011960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.011900    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-home\" (UniqueName: \"kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-home\") pod \"818acd09-3afc-4d37-8443-cb3e4e45550b\" (UID: \"818acd09-3afc-4d37-8443-cb3e4e45550b\") "
Oct 30 22:56:33.011960 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.011945    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-model-cache\") pod \"818acd09-3afc-4d37-8443-cb3e4e45550b\" (UID: \"818acd09-3afc-4d37-8443-cb3e4e45550b\") "
Oct 30 22:56:33.012272 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.012085    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/818acd09-3afc-4d37-8443-cb3e4e45550b-kube-api-access-qkttb\" (UniqueName: \"kubernetes.io/projected/818acd09-3afc-4d37-8443-cb3e4e45550b-kube-api-access-qkttb\") pod \"818acd09-3afc-4d37-8443-cb3e4e45550b\" (UID: \"818acd09-3afc-4d37-8443-cb3e4e45550b\") "
Oct 30 22:56:33.012272 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.012139    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-home" pod "818acd09-3afc-4d37-8443-cb3e4e45550b" (UID: "818acd09-3afc-4d37-8443-cb3e4e45550b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:33.012272 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.012176    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-tmp-dir\") pod \"818acd09-3afc-4d37-8443-cb3e4e45550b\" (UID: \"818acd09-3afc-4d37-8443-cb3e4e45550b\") "
Oct 30 22:56:33.012272 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.012201    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-model-cache" pod "818acd09-3afc-4d37-8443-cb3e4e45550b" (UID: "818acd09-3afc-4d37-8443-cb3e4e45550b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:33.012786 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.012437    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-tmp-dir" pod "818acd09-3afc-4d37-8443-cb3e4e45550b" (UID: "818acd09-3afc-4d37-8443-cb3e4e45550b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:33.013133 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.013107    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:33.013208 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.013138    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:33.013208 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.013156    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:33.014405 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.014380    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-dshm" pod "818acd09-3afc-4d37-8443-cb3e4e45550b" (UID: "818acd09-3afc-4d37-8443-cb3e4e45550b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:33.014555 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.014533    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/818acd09-3afc-4d37-8443-cb3e4e45550b-tls-certs" pod "818acd09-3afc-4d37-8443-cb3e4e45550b" (UID: "818acd09-3afc-4d37-8443-cb3e4e45550b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:33.014614 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.014551    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/818acd09-3afc-4d37-8443-cb3e4e45550b-kube-api-access-qkttb" pod "818acd09-3afc-4d37-8443-cb3e4e45550b" (UID: "818acd09-3afc-4d37-8443-cb3e4e45550b"). InnerVolumeSpecName "kube-api-access-qkttb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:33.115033 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.114938    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkttb\" (UniqueName: \"kubernetes.io/projected/818acd09-3afc-4d37-8443-cb3e4e45550b-kube-api-access-qkttb\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:33.115033 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.114977    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/818acd09-3afc-4d37-8443-cb3e4e45550b-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:33.115033 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.114993    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/818acd09-3afc-4d37-8443-cb3e4e45550b-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:33.488861 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.488767    4889 generic.go:356] "Generic (PLEG): container finished" podID="818acd09-3afc-4d37-8443-cb3e4e45550b" containerID="e94ecd6016f4cf70d4660f75b443ca4cc111c67d5316d0a3b9dd6bc3532d7132" exitCode=0
Oct 30 22:56:33.488861 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.488827    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"
Oct 30 22:56:33.488861 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.488847    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km" event={"ID":"818acd09-3afc-4d37-8443-cb3e4e45550b","Type":"ContainerDied","Data":"e94ecd6016f4cf70d4660f75b443ca4cc111c67d5316d0a3b9dd6bc3532d7132"}
Oct 30 22:56:33.489147 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.488891    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km" event={"ID":"818acd09-3afc-4d37-8443-cb3e4e45550b","Type":"ContainerDied","Data":"9061686167f0b525c36d4b87fdb4e12db6b574d4fc46282d6aded039956dd085"}
Oct 30 22:56:33.489147 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.488915    4889 scope.go:122] "RemoveContainer" containerID="e94ecd6016f4cf70d4660f75b443ca4cc111c67d5316d0a3b9dd6bc3532d7132"
Oct 30 22:56:33.503210 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.503186    4889 scope.go:122] "RemoveContainer" containerID="e94ecd6016f4cf70d4660f75b443ca4cc111c67d5316d0a3b9dd6bc3532d7132"
Oct 30 22:56:33.503667 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:56:33.503635    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e94ecd6016f4cf70d4660f75b443ca4cc111c67d5316d0a3b9dd6bc3532d7132\": container with ID starting with e94ecd6016f4cf70d4660f75b443ca4cc111c67d5316d0a3b9dd6bc3532d7132 not found: ID does not exist" containerID="e94ecd6016f4cf70d4660f75b443ca4cc111c67d5316d0a3b9dd6bc3532d7132"
Oct 30 22:56:33.503780 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.503678    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e94ecd6016f4cf70d4660f75b443ca4cc111c67d5316d0a3b9dd6bc3532d7132"} err="failed to get container status \"e94ecd6016f4cf70d4660f75b443ca4cc111c67d5316d0a3b9dd6bc3532d7132\": rpc error: code = NotFound desc = could not find container \"e94ecd6016f4cf70d4660f75b443ca4cc111c67d5316d0a3b9dd6bc3532d7132\": container with ID starting with e94ecd6016f4cf70d4660f75b443ca4cc111c67d5316d0a3b9dd6bc3532d7132 not found: ID does not exist"
Oct 30 22:56:33.549171 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.549129    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"]
Oct 30 22:56:33.555958 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.555917    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:33.556434 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.556411    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-510f80/auth-test-model-510f80-kserve-67d6f5b849-5g7km"]
Oct 30 22:56:33.631566 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.631528    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh"]
Oct 30 22:56:33.632373 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.632356    4889 cpu_manager.go:414] "containerMap: removing container" podUID="818acd09-3afc-4d37-8443-cb3e4e45550b" containerName="main"
Oct 30 22:56:33.632373 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.632373    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="818acd09-3afc-4d37-8443-cb3e4e45550b" containerName="main"
Oct 30 22:56:33.632513 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.632504    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="818acd09-3afc-4d37-8443-cb3e4e45550b" containerName="main"
Oct 30 22:56:33.637752 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.637727    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh"
Oct 30 22:56:33.654278 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.654203    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f88979\""
Oct 30 22:56:33.654479 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.654451    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f88979-dockercfg-zwtd8\""
Oct 30 22:56:33.657544 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.657517    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh"]
Oct 30 22:56:33.680533 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.680487    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv"]
Oct 30 22:56:33.686678 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.686646    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv"
Oct 30 22:56:33.701677 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.701636    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv"]
Oct 30 22:56:33.727500 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.727456    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b7bae51-8941-42f0-93a1-0a20637b3b75-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh\" (UID: \"7b7bae51-8941-42f0-93a1-0a20637b3b75\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh"
Oct 30 22:56:33.727697 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.727646    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzwqv\" (UniqueName: \"kubernetes.io/projected/e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9-kube-api-access-jzwqv\") pod \"payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv\" (UID: \"e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv"
Oct 30 22:56:33.727697 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.727670    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv\" (UID: \"e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv"
Oct 30 22:56:33.727867 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.727747    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7cgc\" (UniqueName: \"kubernetes.io/projected/7b7bae51-8941-42f0-93a1-0a20637b3b75-kube-api-access-t7cgc\") pod \"payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh\" (UID: \"7b7bae51-8941-42f0-93a1-0a20637b3b75\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh"
Oct 30 22:56:33.738749 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.738701    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:33.842252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.829173    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7cgc\" (UniqueName: \"kubernetes.io/projected/7b7bae51-8941-42f0-93a1-0a20637b3b75-kube-api-access-t7cgc\") pod \"payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh\" (UID: \"7b7bae51-8941-42f0-93a1-0a20637b3b75\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh"
Oct 30 22:56:33.842252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.829313    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b7bae51-8941-42f0-93a1-0a20637b3b75-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh\" (UID: \"7b7bae51-8941-42f0-93a1-0a20637b3b75\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh"
Oct 30 22:56:33.842252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.829514    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzwqv\" (UniqueName: \"kubernetes.io/projected/e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9-kube-api-access-jzwqv\") pod \"payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv\" (UID: \"e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv"
Oct 30 22:56:33.842252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.829549    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv\" (UID: \"e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv"
Oct 30 22:56:33.842252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.830551    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv\" (UID: \"e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv"
Oct 30 22:56:33.842252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.831658    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b7bae51-8941-42f0-93a1-0a20637b3b75-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh\" (UID: \"7b7bae51-8941-42f0-93a1-0a20637b3b75\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh"
Oct 30 22:56:33.856256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.856021    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7cgc\" (UniqueName: \"kubernetes.io/projected/7b7bae51-8941-42f0-93a1-0a20637b3b75-kube-api-access-t7cgc\") pod \"payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh\" (UID: \"7b7bae51-8941-42f0-93a1-0a20637b3b75\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh"
Oct 30 22:56:33.866511 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.866118    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzwqv\" (UniqueName: \"kubernetes.io/projected/e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9-kube-api-access-jzwqv\") pod \"payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv\" (UID: \"e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv"
Oct 30 22:56:33.954084 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:33.954048    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh"
Oct 30 22:56:34.038820 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:34.038468    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv"
Oct 30 22:56:34.296326 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:56:34.296294    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b7bae51_8941_42f0_93a1_0a20637b3b75.slice/crio-997c7e1675ac8ddfc238c7605457067c1d124f91044bd69b6055e66a11f40820 WatchSource:0}: Error finding container 997c7e1675ac8ddfc238c7605457067c1d124f91044bd69b6055e66a11f40820: Status 404 returned error can't find the container with id 997c7e1675ac8ddfc238c7605457067c1d124f91044bd69b6055e66a11f40820
Oct 30 22:56:34.297277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:34.297250    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh"]
Oct 30 22:56:34.389629 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:56:34.389518    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a9064e_4af4_4e01_8c3f_d7e58d78c5a9.slice/crio-6a7c83a8111825db21e6d77838d2efd28d1d3513f838dd8d2c7018f1bed8944b WatchSource:0}: Error finding container 6a7c83a8111825db21e6d77838d2efd28d1d3513f838dd8d2c7018f1bed8944b: Status 404 returned error can't find the container with id 6a7c83a8111825db21e6d77838d2efd28d1d3513f838dd8d2c7018f1bed8944b
Oct 30 22:56:34.390330 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:34.390285    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv"]
Oct 30 22:56:34.506396 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:34.506348    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv" event={"ID":"e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9","Type":"ContainerStarted","Data":"6a7c83a8111825db21e6d77838d2efd28d1d3513f838dd8d2c7018f1bed8944b"}
Oct 30 22:56:34.508741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:34.508700    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh" event={"ID":"7b7bae51-8941-42f0-93a1-0a20637b3b75","Type":"ContainerStarted","Data":"b1c9192cb7a4940d27bad722beffdb1eb911d51bd6f72f07d304100d0743eebb"}
Oct 30 22:56:34.508936 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:34.508768    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh" event={"ID":"7b7bae51-8941-42f0-93a1-0a20637b3b75","Type":"ContainerStarted","Data":"997c7e1675ac8ddfc238c7605457067c1d124f91044bd69b6055e66a11f40820"}
Oct 30 22:56:34.508936 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:34.508799    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh"
Oct 30 22:56:34.521069 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:34.521034    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:34.552842 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:34.552773    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh" podStartSLOduration=1.55275713 podStartE2EDuration="1.55275713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:56:34.551065227 +0000 UTC m=+2979.907431542" watchObservedRunningTime="2026-10-30 22:56:34.55275713 +0000 UTC m=+2979.909123442"
Oct 30 22:56:34.779800 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:34.779765    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="818acd09-3afc-4d37-8443-cb3e4e45550b" path="/var/lib/kubelet/pods/818acd09-3afc-4d37-8443-cb3e4e45550b/volumes"
Oct 30 22:56:35.187929 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:35.187827    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2" podUID="7d175091-47d2-43ad-b201-1b7c0d8b82a9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:56:35.521559 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:35.521521    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv" event={"ID":"e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9","Type":"ContainerStarted","Data":"3edc0df224aa5dbdf84319a03fbc3baa81fb64fcc9064e11c8064bf882f8fa51"}
Oct 30 22:56:35.521794 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:35.521642    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv"
Oct 30 22:56:35.565515 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:35.565446    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv" podStartSLOduration=2.565425346 podStartE2EDuration="2.565425346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:56:35.563851401 +0000 UTC m=+2980.920217716" watchObservedRunningTime="2026-10-30 22:56:35.565425346 +0000 UTC m=+2980.921791662"
Oct 30 22:56:36.216917 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:36.216878    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:36.220983 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:36.220942    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg" podUID="e935aad1-ec9a-45e4-b4d4-31d6b84950a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:56:37.894933 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:37.894891    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:39.540837 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:39.540780    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:40.307201 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:56:40.307164    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:56:41.479079 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:41.479008    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:42.245874 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.245818    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b" podUID="ca61e410-bf75-487e-b37a-333a1141ffb6" containerName="main" containerID="cri-o://104c0e2d926ff6d194443d761396827155528998e02ee40ccc137a016a324de7" gracePeriod=45
Oct 30 22:56:42.565643 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.565610    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"
Oct 30 22:56:42.583658 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.583623    4889 generic.go:356] "Generic (PLEG): container finished" podID="ca61e410-bf75-487e-b37a-333a1141ffb6" containerID="104c0e2d926ff6d194443d761396827155528998e02ee40ccc137a016a324de7" exitCode=0
Oct 30 22:56:42.583839 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.583670    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"
Oct 30 22:56:42.583839 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.583682    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b" event={"ID":"ca61e410-bf75-487e-b37a-333a1141ffb6","Type":"ContainerDied","Data":"104c0e2d926ff6d194443d761396827155528998e02ee40ccc137a016a324de7"}
Oct 30 22:56:42.583839 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.583724    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b" event={"ID":"ca61e410-bf75-487e-b37a-333a1141ffb6","Type":"ContainerDied","Data":"aefb3613f7562f8d9dbc56d4ee69368b056e066c645f23b125b0ab0649f89fb2"}
Oct 30 22:56:42.583839 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.583744    4889 scope.go:122] "RemoveContainer" containerID="104c0e2d926ff6d194443d761396827155528998e02ee40ccc137a016a324de7"
Oct 30 22:56:42.594958 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.594925    4889 scope.go:122] "RemoveContainer" containerID="104c0e2d926ff6d194443d761396827155528998e02ee40ccc137a016a324de7"
Oct 30 22:56:42.595404 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:56:42.595374    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"104c0e2d926ff6d194443d761396827155528998e02ee40ccc137a016a324de7\": container with ID starting with 104c0e2d926ff6d194443d761396827155528998e02ee40ccc137a016a324de7 not found: ID does not exist" containerID="104c0e2d926ff6d194443d761396827155528998e02ee40ccc137a016a324de7"
Oct 30 22:56:42.595528 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.595414    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"104c0e2d926ff6d194443d761396827155528998e02ee40ccc137a016a324de7"} err="failed to get container status \"104c0e2d926ff6d194443d761396827155528998e02ee40ccc137a016a324de7\": rpc error: code = NotFound desc = could not find container \"104c0e2d926ff6d194443d761396827155528998e02ee40ccc137a016a324de7\": container with ID starting with 104c0e2d926ff6d194443d761396827155528998e02ee40ccc137a016a324de7 not found: ID does not exist"
Oct 30 22:56:42.680738 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.680695    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-home\" (UniqueName: \"kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-home\") pod \"ca61e410-bf75-487e-b37a-333a1141ffb6\" (UID: \"ca61e410-bf75-487e-b37a-333a1141ffb6\") "
Oct 30 22:56:42.680935 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.680820    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-model-cache\") pod \"ca61e410-bf75-487e-b37a-333a1141ffb6\" (UID: \"ca61e410-bf75-487e-b37a-333a1141ffb6\") "
Oct 30 22:56:42.681057 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.680965    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca61e410-bf75-487e-b37a-333a1141ffb6-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca61e410-bf75-487e-b37a-333a1141ffb6-tls-certs\") pod \"ca61e410-bf75-487e-b37a-333a1141ffb6\" (UID: \"ca61e410-bf75-487e-b37a-333a1141ffb6\") "
Oct 30 22:56:42.681057 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.681009    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-dshm\") pod \"ca61e410-bf75-487e-b37a-333a1141ffb6\" (UID: \"ca61e410-bf75-487e-b37a-333a1141ffb6\") "
Oct 30 22:56:42.681140 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.681061    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-home" pod "ca61e410-bf75-487e-b37a-333a1141ffb6" (UID: "ca61e410-bf75-487e-b37a-333a1141ffb6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:42.681140 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.681082    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-tmp-dir\") pod \"ca61e410-bf75-487e-b37a-333a1141ffb6\" (UID: \"ca61e410-bf75-487e-b37a-333a1141ffb6\") "
Oct 30 22:56:42.681140 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.681110    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-model-cache" pod "ca61e410-bf75-487e-b37a-333a1141ffb6" (UID: "ca61e410-bf75-487e-b37a-333a1141ffb6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:42.681321 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.681154    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca61e410-bf75-487e-b37a-333a1141ffb6-kube-api-access-5hpv4\" (UniqueName: \"kubernetes.io/projected/ca61e410-bf75-487e-b37a-333a1141ffb6-kube-api-access-5hpv4\") pod \"ca61e410-bf75-487e-b37a-333a1141ffb6\" (UID: \"ca61e410-bf75-487e-b37a-333a1141ffb6\") "
Oct 30 22:56:42.681394 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.681366    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-tmp-dir" pod "ca61e410-bf75-487e-b37a-333a1141ffb6" (UID: "ca61e410-bf75-487e-b37a-333a1141ffb6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:42.681918 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.681858    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:42.681918 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.681887    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:42.681918 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.681900    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:42.683688 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.683644    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca61e410-bf75-487e-b37a-333a1141ffb6-tls-certs" pod "ca61e410-bf75-487e-b37a-333a1141ffb6" (UID: "ca61e410-bf75-487e-b37a-333a1141ffb6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:42.683688 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.683683    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-dshm" pod "ca61e410-bf75-487e-b37a-333a1141ffb6" (UID: "ca61e410-bf75-487e-b37a-333a1141ffb6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:42.683955 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.683929    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca61e410-bf75-487e-b37a-333a1141ffb6-kube-api-access-5hpv4" pod "ca61e410-bf75-487e-b37a-333a1141ffb6" (UID: "ca61e410-bf75-487e-b37a-333a1141ffb6"). InnerVolumeSpecName "kube-api-access-5hpv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:42.783596 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.783507    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hpv4\" (UniqueName: \"kubernetes.io/projected/ca61e410-bf75-487e-b37a-333a1141ffb6-kube-api-access-5hpv4\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:42.783596 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.783550    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca61e410-bf75-487e-b37a-333a1141ffb6-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:42.783596 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.783565    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca61e410-bf75-487e-b37a-333a1141ffb6-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:42.930560 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.930519    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"]
Oct 30 22:56:42.935363 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:42.935329    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-3cd77f/sub-test-model-3cd77f-kserve-7bb498ffcc-xlh6b"]
Oct 30 22:56:43.135293 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:43.135192    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:44.776981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:44.776948    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca61e410-bf75-487e-b37a-333a1141ffb6" path="/var/lib/kubelet/pods/ca61e410-bf75-487e-b37a-333a1141ffb6/volumes"
Oct 30 22:56:44.943633 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:44.943594    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:45.187755 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:45.187662    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2" podUID="7d175091-47d2-43ad-b201-1b7c0d8b82a9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:56:45.525713 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:45.525679    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh"
Oct 30 22:56:46.221211 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:46.221167    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg" podUID="e935aad1-ec9a-45e4-b4d4-31d6b84950a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:56:46.532485 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:46.532454    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv"
Oct 30 22:56:46.730663 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:46.730624    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:47.052911 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.052872    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"]
Oct 30 22:56:47.053883 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.053858    4889 cpu_manager.go:414] "containerMap: removing container" podUID="ca61e410-bf75-487e-b37a-333a1141ffb6" containerName="main"
Oct 30 22:56:47.053883 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.053881    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca61e410-bf75-487e-b37a-333a1141ffb6" containerName="main"
Oct 30 22:56:47.054097 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.054084    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca61e410-bf75-487e-b37a-333a1141ffb6" containerName="main"
Oct 30 22:56:47.064082 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.064042    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"
Oct 30 22:56:47.086763 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.086718    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"]
Oct 30 22:56:47.092630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.092588    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-6f2eba\"/\"rate-test-model-6f2eba-kserve-self-signed-certs\""
Oct 30 22:56:47.093191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.093123    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-6f2eba\"/\"default-dockercfg-cfdgd\""
Oct 30 22:56:47.095329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.095301    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-6f2eba\"/\"openshift-service-ca.crt\""
Oct 30 22:56:47.095329 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.095341    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-6f2eba\"/\"kube-root-ca.crt\""
Oct 30 22:56:47.148653 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.148609    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-dshm\") pod \"rate-test-model-6f2eba-kserve-855bb4998f-bprqf\" (UID: \"2eb013be-cbf1-4d26-9393-00c6498157de\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"
Oct 30 22:56:47.148653 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.148657    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-home\") pod \"rate-test-model-6f2eba-kserve-855bb4998f-bprqf\" (UID: \"2eb013be-cbf1-4d26-9393-00c6498157de\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"
Oct 30 22:56:47.148904 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.148790    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eb013be-cbf1-4d26-9393-00c6498157de-tls-certs\") pod \"rate-test-model-6f2eba-kserve-855bb4998f-bprqf\" (UID: \"2eb013be-cbf1-4d26-9393-00c6498157de\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"
Oct 30 22:56:47.148952 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.148931    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-tmp-dir\") pod \"rate-test-model-6f2eba-kserve-855bb4998f-bprqf\" (UID: \"2eb013be-cbf1-4d26-9393-00c6498157de\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"
Oct 30 22:56:47.149040 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.149025    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-model-cache\") pod \"rate-test-model-6f2eba-kserve-855bb4998f-bprqf\" (UID: \"2eb013be-cbf1-4d26-9393-00c6498157de\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"
Oct 30 22:56:47.149113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.149099    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qgrt\" (UniqueName: \"kubernetes.io/projected/2eb013be-cbf1-4d26-9393-00c6498157de-kube-api-access-4qgrt\") pod \"rate-test-model-6f2eba-kserve-855bb4998f-bprqf\" (UID: \"2eb013be-cbf1-4d26-9393-00c6498157de\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"
Oct 30 22:56:47.183940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.183853    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:47.251059 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.251012    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eb013be-cbf1-4d26-9393-00c6498157de-tls-certs\") pod \"rate-test-model-6f2eba-kserve-855bb4998f-bprqf\" (UID: \"2eb013be-cbf1-4d26-9393-00c6498157de\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"
Oct 30 22:56:47.251615 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.251140    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-tmp-dir\") pod \"rate-test-model-6f2eba-kserve-855bb4998f-bprqf\" (UID: \"2eb013be-cbf1-4d26-9393-00c6498157de\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"
Oct 30 22:56:47.251615 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.251285    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-model-cache\") pod \"rate-test-model-6f2eba-kserve-855bb4998f-bprqf\" (UID: \"2eb013be-cbf1-4d26-9393-00c6498157de\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"
Oct 30 22:56:47.251615 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.251366    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qgrt\" (UniqueName: \"kubernetes.io/projected/2eb013be-cbf1-4d26-9393-00c6498157de-kube-api-access-4qgrt\") pod \"rate-test-model-6f2eba-kserve-855bb4998f-bprqf\" (UID: \"2eb013be-cbf1-4d26-9393-00c6498157de\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"
Oct 30 22:56:47.251615 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.251475    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-dshm\") pod \"rate-test-model-6f2eba-kserve-855bb4998f-bprqf\" (UID: \"2eb013be-cbf1-4d26-9393-00c6498157de\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"
Oct 30 22:56:47.251615 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.251503    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-home\") pod \"rate-test-model-6f2eba-kserve-855bb4998f-bprqf\" (UID: \"2eb013be-cbf1-4d26-9393-00c6498157de\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"
Oct 30 22:56:47.251956 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.251733    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-tmp-dir\") pod \"rate-test-model-6f2eba-kserve-855bb4998f-bprqf\" (UID: \"2eb013be-cbf1-4d26-9393-00c6498157de\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"
Oct 30 22:56:47.251956 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.251755    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-model-cache\") pod \"rate-test-model-6f2eba-kserve-855bb4998f-bprqf\" (UID: \"2eb013be-cbf1-4d26-9393-00c6498157de\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"
Oct 30 22:56:47.251956 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.251930    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-home\") pod \"rate-test-model-6f2eba-kserve-855bb4998f-bprqf\" (UID: \"2eb013be-cbf1-4d26-9393-00c6498157de\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"
Oct 30 22:56:47.253866 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.253842    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-dshm\") pod \"rate-test-model-6f2eba-kserve-855bb4998f-bprqf\" (UID: \"2eb013be-cbf1-4d26-9393-00c6498157de\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"
Oct 30 22:56:47.254169 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.254147    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eb013be-cbf1-4d26-9393-00c6498157de-tls-certs\") pod \"rate-test-model-6f2eba-kserve-855bb4998f-bprqf\" (UID: \"2eb013be-cbf1-4d26-9393-00c6498157de\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"
Oct 30 22:56:47.273239 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.273192    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qgrt\" (UniqueName: \"kubernetes.io/projected/2eb013be-cbf1-4d26-9393-00c6498157de-kube-api-access-4qgrt\") pod \"rate-test-model-6f2eba-kserve-855bb4998f-bprqf\" (UID: \"2eb013be-cbf1-4d26-9393-00c6498157de\") " pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"
Oct 30 22:56:47.395428 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.395389    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"
Oct 30 22:56:47.707180 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:56:47.707107    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2eb013be_cbf1_4d26_9393_00c6498157de.slice/crio-05839fcfc62313ea2a3946d3b91c82abfff70831f1dc97aa8398ce71f8048aa5 WatchSource:0}: Error finding container 05839fcfc62313ea2a3946d3b91c82abfff70831f1dc97aa8398ce71f8048aa5: Status 404 returned error can't find the container with id 05839fcfc62313ea2a3946d3b91c82abfff70831f1dc97aa8398ce71f8048aa5
Oct 30 22:56:47.707946 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:47.707920    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"]
Oct 30 22:56:48.639777 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:48.639737    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf" event={"ID":"2eb013be-cbf1-4d26-9393-00c6498157de","Type":"ContainerStarted","Data":"365ca75a9130796f64070d9ff210f19019fd80f11a753f9bb32fe6e4bca1e8bb"}
Oct 30 22:56:48.639777 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:48.639783    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf" event={"ID":"2eb013be-cbf1-4d26-9393-00c6498157de","Type":"ContainerStarted","Data":"05839fcfc62313ea2a3946d3b91c82abfff70831f1dc97aa8398ce71f8048aa5"}
Oct 30 22:56:48.686474 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:48.686416    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf" podStartSLOduration=1.686401791 podStartE2EDuration="1.686401791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:56:48.683561015 +0000 UTC m=+2994.039927330" watchObservedRunningTime="2026-10-30 22:56:48.686401791 +0000 UTC m=+2994.042768105"
Oct 30 22:56:48.983682 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:48.983601    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"]
Oct 30 22:56:48.983853 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:48.983810    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292" podUID="014efbd4-b548-4bf9-8b36-50285f54073f" containerName="istio-proxy" containerID="cri-o://15f7d9d87e68b2cb5758593cc553a87bffcfd33f72749e377ae635222f7d6bf7" gracePeriod=30
Oct 30 22:56:49.396059 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:49.396024    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"
Oct 30 22:56:49.522666 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:49.522623    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:50.251943 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:50.251893    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:50.844054 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:50.844015    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:52.031868 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:52.031810    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:53.236672 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.236645    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg"
Oct 30 22:56:53.276809 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.276783    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2"
Oct 30 22:56:53.339095 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.339053    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e935aad1-ec9a-45e4-b4d4-31d6b84950a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e935aad1-ec9a-45e4-b4d4-31d6b84950a1-plugins-config-volume\") pod \"e935aad1-ec9a-45e4-b4d4-31d6b84950a1\" (UID: \"e935aad1-ec9a-45e4-b4d4-31d6b84950a1\") "
Oct 30 22:56:53.339310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.339170    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d175091-47d2-43ad-b201-1b7c0d8b82a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d175091-47d2-43ad-b201-1b7c0d8b82a9-plugins-config-volume\") pod \"7d175091-47d2-43ad-b201-1b7c0d8b82a9\" (UID: \"7d175091-47d2-43ad-b201-1b7c0d8b82a9\") "
Oct 30 22:56:53.339310 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.339276    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e935aad1-ec9a-45e4-b4d4-31d6b84950a1-kube-api-access-qzx4f\" (UniqueName: \"kubernetes.io/projected/e935aad1-ec9a-45e4-b4d4-31d6b84950a1-kube-api-access-qzx4f\") pod \"e935aad1-ec9a-45e4-b4d4-31d6b84950a1\" (UID: \"e935aad1-ec9a-45e4-b4d4-31d6b84950a1\") "
Oct 30 22:56:53.339585 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.339556    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e935aad1-ec9a-45e4-b4d4-31d6b84950a1-plugins-config-volume" pod "e935aad1-ec9a-45e4-b4d4-31d6b84950a1" (UID: "e935aad1-ec9a-45e4-b4d4-31d6b84950a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:56:53.339662 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.339558    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d175091-47d2-43ad-b201-1b7c0d8b82a9-plugins-config-volume" pod "7d175091-47d2-43ad-b201-1b7c0d8b82a9" (UID: "7d175091-47d2-43ad-b201-1b7c0d8b82a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:56:53.339746 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.339724    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d175091-47d2-43ad-b201-1b7c0d8b82a9-kube-api-access-c47jw\" (UniqueName: \"kubernetes.io/projected/7d175091-47d2-43ad-b201-1b7c0d8b82a9-kube-api-access-c47jw\") pod \"7d175091-47d2-43ad-b201-1b7c0d8b82a9\" (UID: \"7d175091-47d2-43ad-b201-1b7c0d8b82a9\") "
Oct 30 22:56:53.340779 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.340752    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e935aad1-ec9a-45e4-b4d4-31d6b84950a1-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:53.340918 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.340784    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d175091-47d2-43ad-b201-1b7c0d8b82a9-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:53.342019 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.341994    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e935aad1-ec9a-45e4-b4d4-31d6b84950a1-kube-api-access-qzx4f" pod "e935aad1-ec9a-45e4-b4d4-31d6b84950a1" (UID: "e935aad1-ec9a-45e4-b4d4-31d6b84950a1"). InnerVolumeSpecName "kube-api-access-qzx4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:53.342529 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.342509    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d175091-47d2-43ad-b201-1b7c0d8b82a9-kube-api-access-c47jw" pod "7d175091-47d2-43ad-b201-1b7c0d8b82a9" (UID: "7d175091-47d2-43ad-b201-1b7c0d8b82a9"). InnerVolumeSpecName "kube-api-access-c47jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:53.442240 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.442133    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzx4f\" (UniqueName: \"kubernetes.io/projected/e935aad1-ec9a-45e4-b4d4-31d6b84950a1-kube-api-access-qzx4f\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:53.442240 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.442166    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c47jw\" (UniqueName: \"kubernetes.io/projected/7d175091-47d2-43ad-b201-1b7c0d8b82a9-kube-api-access-c47jw\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:53.692527 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.692435    4889 generic.go:356] "Generic (PLEG): container finished" podID="e935aad1-ec9a-45e4-b4d4-31d6b84950a1" containerID="7aa54b2d5342eedd14e9e50f6079d7bc370bda3a857f40f4521a57e0c8c2de0c" exitCode=0
Oct 30 22:56:53.692527 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.692485    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg"
Oct 30 22:56:53.692527 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.692496    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg" event={"ID":"e935aad1-ec9a-45e4-b4d4-31d6b84950a1","Type":"ContainerDied","Data":"7aa54b2d5342eedd14e9e50f6079d7bc370bda3a857f40f4521a57e0c8c2de0c"}
Oct 30 22:56:53.692823 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.692558    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg" event={"ID":"e935aad1-ec9a-45e4-b4d4-31d6b84950a1","Type":"ContainerDied","Data":"b4b5eb5627ef5ef876cde108b7e82edb960c9bb5100ae7ff96bced9594c13aa8"}
Oct 30 22:56:53.692823 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.692583    4889 scope.go:122] "RemoveContainer" containerID="7aa54b2d5342eedd14e9e50f6079d7bc370bda3a857f40f4521a57e0c8c2de0c"
Oct 30 22:56:53.694503 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.694477    4889 generic.go:356] "Generic (PLEG): container finished" podID="7d175091-47d2-43ad-b201-1b7c0d8b82a9" containerID="6afbca25923bc23cabb77349037c6545edb90c4ce30a263ece94aeb7f32ef1ee" exitCode=0
Oct 30 22:56:53.694709 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.694541    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2" event={"ID":"7d175091-47d2-43ad-b201-1b7c0d8b82a9","Type":"ContainerDied","Data":"6afbca25923bc23cabb77349037c6545edb90c4ce30a263ece94aeb7f32ef1ee"}
Oct 30 22:56:53.694709 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.694702    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2"
Oct 30 22:56:53.694831 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.694744    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2" event={"ID":"7d175091-47d2-43ad-b201-1b7c0d8b82a9","Type":"ContainerDied","Data":"81e3ef97c83ed9569f7e05a0b9f5e010c6530f72dc1cbe70fc33622035e09be0"}
Oct 30 22:56:53.707445 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.707421    4889 scope.go:122] "RemoveContainer" containerID="7aa54b2d5342eedd14e9e50f6079d7bc370bda3a857f40f4521a57e0c8c2de0c"
Oct 30 22:56:53.707735 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:56:53.707715    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aa54b2d5342eedd14e9e50f6079d7bc370bda3a857f40f4521a57e0c8c2de0c\": container with ID starting with 7aa54b2d5342eedd14e9e50f6079d7bc370bda3a857f40f4521a57e0c8c2de0c not found: ID does not exist" containerID="7aa54b2d5342eedd14e9e50f6079d7bc370bda3a857f40f4521a57e0c8c2de0c"
Oct 30 22:56:53.707795 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.707744    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aa54b2d5342eedd14e9e50f6079d7bc370bda3a857f40f4521a57e0c8c2de0c"} err="failed to get container status \"7aa54b2d5342eedd14e9e50f6079d7bc370bda3a857f40f4521a57e0c8c2de0c\": rpc error: code = NotFound desc = could not find container \"7aa54b2d5342eedd14e9e50f6079d7bc370bda3a857f40f4521a57e0c8c2de0c\": container with ID starting with 7aa54b2d5342eedd14e9e50f6079d7bc370bda3a857f40f4521a57e0c8c2de0c not found: ID does not exist"
Oct 30 22:56:53.707795 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.707766    4889 scope.go:122] "RemoveContainer" containerID="6afbca25923bc23cabb77349037c6545edb90c4ce30a263ece94aeb7f32ef1ee"
Oct 30 22:56:53.718659 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.718637    4889 scope.go:122] "RemoveContainer" containerID="6afbca25923bc23cabb77349037c6545edb90c4ce30a263ece94aeb7f32ef1ee"
Oct 30 22:56:53.719003 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:56:53.718980    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6afbca25923bc23cabb77349037c6545edb90c4ce30a263ece94aeb7f32ef1ee\": container with ID starting with 6afbca25923bc23cabb77349037c6545edb90c4ce30a263ece94aeb7f32ef1ee not found: ID does not exist" containerID="6afbca25923bc23cabb77349037c6545edb90c4ce30a263ece94aeb7f32ef1ee"
Oct 30 22:56:53.719091 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.719010    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6afbca25923bc23cabb77349037c6545edb90c4ce30a263ece94aeb7f32ef1ee"} err="failed to get container status \"6afbca25923bc23cabb77349037c6545edb90c4ce30a263ece94aeb7f32ef1ee\": rpc error: code = NotFound desc = could not find container \"6afbca25923bc23cabb77349037c6545edb90c4ce30a263ece94aeb7f32ef1ee\": container with ID starting with 6afbca25923bc23cabb77349037c6545edb90c4ce30a263ece94aeb7f32ef1ee not found: ID does not exist"
Oct 30 22:56:53.734776 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.734741    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:53.756136 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.756104    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2"]
Oct 30 22:56:53.760053 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.760022    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c511a3-7c6f5989ff-r6zv2"]
Oct 30 22:56:53.796690 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.796652    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg"]
Oct 30 22:56:53.800477 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:53.800452    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c511a3-5ddf74769-fztjg"]
Oct 30 22:56:54.281889 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.281860    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:56:54.357411 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.357371    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-istio-data\") pod \"014efbd4-b548-4bf9-8b36-50285f54073f\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") "
Oct 30 22:56:54.357611 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.357429    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/014efbd4-b548-4bf9-8b36-50285f54073f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014efbd4-b548-4bf9-8b36-50285f54073f-istiod-ca-cert\") pod \"014efbd4-b548-4bf9-8b36-50285f54073f\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") "
Oct 30 22:56:54.357611 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.357476    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-workload-certs\") pod \"014efbd4-b548-4bf9-8b36-50285f54073f\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") "
Oct 30 22:56:54.357611 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.357514    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-istio-envoy\") pod \"014efbd4-b548-4bf9-8b36-50285f54073f\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") "
Oct 30 22:56:54.357792 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.357657    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-istio-data" pod "014efbd4-b548-4bf9-8b36-50285f54073f" (UID: "014efbd4-b548-4bf9-8b36-50285f54073f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:54.357952 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.357931    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014efbd4-b548-4bf9-8b36-50285f54073f-istiod-ca-cert" pod "014efbd4-b548-4bf9-8b36-50285f54073f" (UID: "014efbd4-b548-4bf9-8b36-50285f54073f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:56:54.358034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.358020    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-credential-socket\") pod \"014efbd4-b548-4bf9-8b36-50285f54073f\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") "
Oct 30 22:56:54.358091 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.358049    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014efbd4-b548-4bf9-8b36-50285f54073f-kube-api-access-wcqjd\" (UniqueName: \"kubernetes.io/projected/014efbd4-b548-4bf9-8b36-50285f54073f-kube-api-access-wcqjd\") pod \"014efbd4-b548-4bf9-8b36-50285f54073f\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") "
Oct 30 22:56:54.358167 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.358150    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-workload-socket\") pod \"014efbd4-b548-4bf9-8b36-50285f54073f\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") "
Oct 30 22:56:54.358245 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.358209    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014efbd4-b548-4bf9-8b36-50285f54073f-istio-token\" (UniqueName: \"kubernetes.io/projected/014efbd4-b548-4bf9-8b36-50285f54073f-istio-token\") pod \"014efbd4-b548-4bf9-8b36-50285f54073f\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") "
Oct 30 22:56:54.358386 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.358366    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/014efbd4-b548-4bf9-8b36-50285f54073f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014efbd4-b548-4bf9-8b36-50285f54073f-istio-podinfo\") pod \"014efbd4-b548-4bf9-8b36-50285f54073f\" (UID: \"014efbd4-b548-4bf9-8b36-50285f54073f\") "
Oct 30 22:56:54.358448 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.358395    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-credential-socket" pod "014efbd4-b548-4bf9-8b36-50285f54073f" (UID: "014efbd4-b548-4bf9-8b36-50285f54073f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:54.358615 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.358593    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-workload-socket" pod "014efbd4-b548-4bf9-8b36-50285f54073f" (UID: "014efbd4-b548-4bf9-8b36-50285f54073f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:54.359399 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.359368    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:54.359546 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.359403    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:54.359546 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.359418    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:54.359546 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.359433    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014efbd4-b548-4bf9-8b36-50285f54073f-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:54.361090 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.361065    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-workload-certs" pod "014efbd4-b548-4bf9-8b36-50285f54073f" (UID: "014efbd4-b548-4bf9-8b36-50285f54073f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:54.362984 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.362961    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-istio-envoy" pod "014efbd4-b548-4bf9-8b36-50285f54073f" (UID: "014efbd4-b548-4bf9-8b36-50285f54073f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:54.364203 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.364174    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014efbd4-b548-4bf9-8b36-50285f54073f-istio-token" pod "014efbd4-b548-4bf9-8b36-50285f54073f" (UID: "014efbd4-b548-4bf9-8b36-50285f54073f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:54.365009 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.364673    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014efbd4-b548-4bf9-8b36-50285f54073f-kube-api-access-wcqjd" pod "014efbd4-b548-4bf9-8b36-50285f54073f" (UID: "014efbd4-b548-4bf9-8b36-50285f54073f"). InnerVolumeSpecName "kube-api-access-wcqjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:54.365383 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.365357    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/014efbd4-b548-4bf9-8b36-50285f54073f-istio-podinfo" pod "014efbd4-b548-4bf9-8b36-50285f54073f" (UID: "014efbd4-b548-4bf9-8b36-50285f54073f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:56:54.461006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.460969    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:54.461006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.461006    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014efbd4-b548-4bf9-8b36-50285f54073f-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:54.461006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.461020    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcqjd\" (UniqueName: \"kubernetes.io/projected/014efbd4-b548-4bf9-8b36-50285f54073f-kube-api-access-wcqjd\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:54.461328 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.461035    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014efbd4-b548-4bf9-8b36-50285f54073f-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:54.461328 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.461050    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014efbd4-b548-4bf9-8b36-50285f54073f-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:56:54.705724 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.705627    4889 generic.go:356] "Generic (PLEG): container finished" podID="014efbd4-b548-4bf9-8b36-50285f54073f" containerID="15f7d9d87e68b2cb5758593cc553a87bffcfd33f72749e377ae635222f7d6bf7" exitCode=0
Oct 30 22:56:54.705724 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.705682    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"
Oct 30 22:56:54.705981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.705714    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292" event={"ID":"014efbd4-b548-4bf9-8b36-50285f54073f","Type":"ContainerDied","Data":"15f7d9d87e68b2cb5758593cc553a87bffcfd33f72749e377ae635222f7d6bf7"}
Oct 30 22:56:54.705981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.705775    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292" event={"ID":"014efbd4-b548-4bf9-8b36-50285f54073f","Type":"ContainerDied","Data":"f3628330c1818c2a6b21102a870c67d4a6fbf2c7082fc39b39dd9edb96292fc8"}
Oct 30 22:56:54.705981 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.705799    4889 scope.go:122] "RemoveContainer" containerID="15f7d9d87e68b2cb5758593cc553a87bffcfd33f72749e377ae635222f7d6bf7"
Oct 30 22:56:54.718057 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.718032    4889 scope.go:122] "RemoveContainer" containerID="15f7d9d87e68b2cb5758593cc553a87bffcfd33f72749e377ae635222f7d6bf7"
Oct 30 22:56:54.718390 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:56:54.718372    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15f7d9d87e68b2cb5758593cc553a87bffcfd33f72749e377ae635222f7d6bf7\": container with ID starting with 15f7d9d87e68b2cb5758593cc553a87bffcfd33f72749e377ae635222f7d6bf7 not found: ID does not exist" containerID="15f7d9d87e68b2cb5758593cc553a87bffcfd33f72749e377ae635222f7d6bf7"
Oct 30 22:56:54.718462 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.718400    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15f7d9d87e68b2cb5758593cc553a87bffcfd33f72749e377ae635222f7d6bf7"} err="failed to get container status \"15f7d9d87e68b2cb5758593cc553a87bffcfd33f72749e377ae635222f7d6bf7\": rpc error: code = NotFound desc = could not find container \"15f7d9d87e68b2cb5758593cc553a87bffcfd33f72749e377ae635222f7d6bf7\": container with ID starting with 15f7d9d87e68b2cb5758593cc553a87bffcfd33f72749e377ae635222f7d6bf7 not found: ID does not exist"
Oct 30 22:56:54.764801 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.764748    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"]
Oct 30 22:56:54.788823 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.788781    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d175091-47d2-43ad-b201-1b7c0d8b82a9" path="/var/lib/kubelet/pods/7d175091-47d2-43ad-b201-1b7c0d8b82a9/volumes"
Oct 30 22:56:54.789296 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.789282    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e935aad1-ec9a-45e4-b4d4-31d6b84950a1" path="/var/lib/kubelet/pods/e935aad1-ec9a-45e4-b4d4-31d6b84950a1/volumes"
Oct 30 22:56:54.789641 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:54.789628    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-hs292"]
Oct 30 22:56:55.462860 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:55.462735    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:55.723150 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:55.723057    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 22:56:55.727684 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:55.727653    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 22:56:56.777406 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:56.777374    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="014efbd4-b548-4bf9-8b36-50285f54073f" path="/var/lib/kubelet/pods/014efbd4-b548-4bf9-8b36-50285f54073f/volumes"
Oct 30 22:56:57.060688 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.060647    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf"]
Oct 30 22:56:57.062747 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.062720    4889 cpu_manager.go:414] "containerMap: removing container" podUID="7d175091-47d2-43ad-b201-1b7c0d8b82a9" containerName="payload-pre-processing"
Oct 30 22:56:57.062861 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.062752    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d175091-47d2-43ad-b201-1b7c0d8b82a9" containerName="payload-pre-processing"
Oct 30 22:56:57.062861 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.062796    4889 cpu_manager.go:414] "containerMap: removing container" podUID="014efbd4-b548-4bf9-8b36-50285f54073f" containerName="istio-proxy"
Oct 30 22:56:57.062861 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.062805    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="014efbd4-b548-4bf9-8b36-50285f54073f" containerName="istio-proxy"
Oct 30 22:56:57.063085 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.062932    4889 cpu_manager.go:414] "containerMap: removing container" podUID="e935aad1-ec9a-45e4-b4d4-31d6b84950a1" containerName="payload-processing"
Oct 30 22:56:57.063085 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.062942    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e935aad1-ec9a-45e4-b4d4-31d6b84950a1" containerName="payload-processing"
Oct 30 22:56:57.063661 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.063625    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="e935aad1-ec9a-45e4-b4d4-31d6b84950a1" containerName="payload-processing"
Oct 30 22:56:57.063789 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.063674    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d175091-47d2-43ad-b201-1b7c0d8b82a9" containerName="payload-pre-processing"
Oct 30 22:56:57.063789 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.063686    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="014efbd4-b548-4bf9-8b36-50285f54073f" containerName="istio-proxy"
Oct 30 22:56:57.073707 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.073676    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf"]
Oct 30 22:56:57.073888 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.073852    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf"
Oct 30 22:56:57.090649 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.090597    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f0a807-serving-cert\""
Oct 30 22:56:57.195947 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.195913    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:57.199455 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.199418    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf\" (UID: \"1c1dcc08-1daf-4112-93d9-036d7dd70f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf"
Oct 30 22:56:57.199455 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.199465    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf\" (UID: \"1c1dcc08-1daf-4112-93d9-036d7dd70f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf"
Oct 30 22:56:57.199708 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.199617    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-metrics-certs\") pod \"maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf\" (UID: \"1c1dcc08-1daf-4112-93d9-036d7dd70f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf"
Oct 30 22:56:57.199708 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.199642    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gqr6\" (UniqueName: \"kubernetes.io/projected/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-kube-api-access-8gqr6\") pod \"maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf\" (UID: \"1c1dcc08-1daf-4112-93d9-036d7dd70f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf"
Oct 30 22:56:57.301514 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.301418    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf\" (UID: \"1c1dcc08-1daf-4112-93d9-036d7dd70f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf"
Oct 30 22:56:57.301514 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.301466    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf\" (UID: \"1c1dcc08-1daf-4112-93d9-036d7dd70f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf"
Oct 30 22:56:57.301733 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.301580    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-metrics-certs\") pod \"maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf\" (UID: \"1c1dcc08-1daf-4112-93d9-036d7dd70f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf"
Oct 30 22:56:57.301733 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.301609    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gqr6\" (UniqueName: \"kubernetes.io/projected/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-kube-api-access-8gqr6\") pod \"maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf\" (UID: \"1c1dcc08-1daf-4112-93d9-036d7dd70f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf"
Oct 30 22:56:57.302212 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.302186    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf\" (UID: \"1c1dcc08-1daf-4112-93d9-036d7dd70f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf"
Oct 30 22:56:57.304152 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.304126    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-metrics-certs\") pod \"maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf\" (UID: \"1c1dcc08-1daf-4112-93d9-036d7dd70f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf"
Oct 30 22:56:57.304261 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.304199    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf\" (UID: \"1c1dcc08-1daf-4112-93d9-036d7dd70f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf"
Oct 30 22:56:57.324111 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.324071    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gqr6\" (UniqueName: \"kubernetes.io/projected/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-kube-api-access-8gqr6\") pod \"maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf\" (UID: \"1c1dcc08-1daf-4112-93d9-036d7dd70f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf"
Oct 30 22:56:57.365361 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.365311    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:57.396395 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.396363    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"
Oct 30 22:56:57.397514 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.397491    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf"
Oct 30 22:56:57.411128 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.411099    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"
Oct 30 22:56:57.541667 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.538591    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:57.727783 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:56:57.727751    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c1dcc08_1daf_4112_93d9_036d7dd70f6c.slice/crio-0e3e22c841fa2252703f2183d96c71922f58743e6632ec1606ffceffa034fe2d WatchSource:0}: Error finding container 0e3e22c841fa2252703f2183d96c71922f58743e6632ec1606ffceffa034fe2d: Status 404 returned error can't find the container with id 0e3e22c841fa2252703f2183d96c71922f58743e6632ec1606ffceffa034fe2d
Oct 30 22:56:57.729479 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.729457    4889 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:56:57.729619 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.729484    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf"]
Oct 30 22:56:57.744915 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:57.744882    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"
Oct 30 22:56:58.744462 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:58.744426    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf" event={"ID":"1c1dcc08-1daf-4112-93d9-036d7dd70f6c","Type":"ContainerStarted","Data":"2f7275f3044311f78eab8b80e48ddf6507a50b3ceef5ff97590a1d7c86c006fe"}
Oct 30 22:56:58.745030 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:58.744485    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf" event={"ID":"1c1dcc08-1daf-4112-93d9-036d7dd70f6c","Type":"ContainerStarted","Data":"0e3e22c841fa2252703f2183d96c71922f58743e6632ec1606ffceffa034fe2d"}
Oct 30 22:56:58.745030 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:58.744524    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf"
Oct 30 22:56:58.792771 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:58.792717    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf" podStartSLOduration=1.4687255289999999 podStartE2EDuration="1.792701287s" totalImagesPullingTime="323.975758ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:56:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:56:58.787874101 +0000 UTC m=+3004.144240409" watchObservedRunningTime="2026-10-30 22:56:58.792701287 +0000 UTC m=+3004.149067602"
Oct 30 22:56:59.102976 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:56:59.102887    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:00.874609 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:00.874563    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:02.199084 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:02.199040    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg"]
Oct 30 22:57:02.199650 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:02.199387    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg" podUID="bdd7407b-f54b-4e2a-8215-1b1a740b1d69" containerName="maas-api" containerID="cri-o://d2b876b0153e506aff6408c123693f5d45dbc5fcb04e3ce848d2cbc3ec69f0ca" gracePeriod=30
Oct 30 22:57:02.320765 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:02.320724    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:02.521075 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:02.521037    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg"
Oct 30 22:57:02.593185 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:02.593135    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-maas-api-tls\") pod \"bdd7407b-f54b-4e2a-8215-1b1a740b1d69\" (UID: \"bdd7407b-f54b-4e2a-8215-1b1a740b1d69\") "
Oct 30 22:57:02.593463 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:02.593283    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-metrics-certs\") pod \"bdd7407b-f54b-4e2a-8215-1b1a740b1d69\" (UID: \"bdd7407b-f54b-4e2a-8215-1b1a740b1d69\") "
Oct 30 22:57:02.593463 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:02.593391    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-openshift-service-ca\") pod \"bdd7407b-f54b-4e2a-8215-1b1a740b1d69\" (UID: \"bdd7407b-f54b-4e2a-8215-1b1a740b1d69\") "
Oct 30 22:57:02.593607 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:02.593588    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-kube-api-access-pgb8l\" (UniqueName: \"kubernetes.io/projected/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-kube-api-access-pgb8l\") pod \"bdd7407b-f54b-4e2a-8215-1b1a740b1d69\" (UID: \"bdd7407b-f54b-4e2a-8215-1b1a740b1d69\") "
Oct 30 22:57:02.595774 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:02.595638    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-openshift-service-ca" pod "bdd7407b-f54b-4e2a-8215-1b1a740b1d69" (UID: "bdd7407b-f54b-4e2a-8215-1b1a740b1d69"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:57:02.597511 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:02.597456    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-metrics-certs" pod "bdd7407b-f54b-4e2a-8215-1b1a740b1d69" (UID: "bdd7407b-f54b-4e2a-8215-1b1a740b1d69"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:57:02.598763 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:02.598736    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-maas-api-tls" pod "bdd7407b-f54b-4e2a-8215-1b1a740b1d69" (UID: "bdd7407b-f54b-4e2a-8215-1b1a740b1d69"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:57:02.598952 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:02.598934    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-kube-api-access-pgb8l" pod "bdd7407b-f54b-4e2a-8215-1b1a740b1d69" (UID: "bdd7407b-f54b-4e2a-8215-1b1a740b1d69"). InnerVolumeSpecName "kube-api-access-pgb8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:02.695885 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:02.695842    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:57:02.695885 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:02.695877    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:57:02.695885 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:02.695886    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:57:02.696256 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:02.695899    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgb8l\" (UniqueName: \"kubernetes.io/projected/bdd7407b-f54b-4e2a-8215-1b1a740b1d69-kube-api-access-pgb8l\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:57:02.815298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:02.810477    4889 generic.go:356] "Generic (PLEG): container finished" podID="bdd7407b-f54b-4e2a-8215-1b1a740b1d69" containerID="d2b876b0153e506aff6408c123693f5d45dbc5fcb04e3ce848d2cbc3ec69f0ca" exitCode=0
Oct 30 22:57:02.815298 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:02.810676    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg"
Oct 30 22:57:02.829641 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:02.829461    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg" event={"ID":"bdd7407b-f54b-4e2a-8215-1b1a740b1d69","Type":"ContainerDied","Data":"d2b876b0153e506aff6408c123693f5d45dbc5fcb04e3ce848d2cbc3ec69f0ca"}
Oct 30 22:57:02.829641 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:02.829506    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg" event={"ID":"bdd7407b-f54b-4e2a-8215-1b1a740b1d69","Type":"ContainerDied","Data":"d9beba1817f3c1ff96ce2c35adb9892e86aeee1127fd3b1238c0323bbaca001b"}
Oct 30 22:57:02.829641 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:02.829563    4889 scope.go:122] "RemoveContainer" containerID="d2b876b0153e506aff6408c123693f5d45dbc5fcb04e3ce848d2cbc3ec69f0ca"
Oct 30 22:57:02.843086 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:02.843063    4889 scope.go:122] "RemoveContainer" containerID="d2b876b0153e506aff6408c123693f5d45dbc5fcb04e3ce848d2cbc3ec69f0ca"
Oct 30 22:57:02.843606 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:57:02.843569    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2b876b0153e506aff6408c123693f5d45dbc5fcb04e3ce848d2cbc3ec69f0ca\": container with ID starting with d2b876b0153e506aff6408c123693f5d45dbc5fcb04e3ce848d2cbc3ec69f0ca not found: ID does not exist" containerID="d2b876b0153e506aff6408c123693f5d45dbc5fcb04e3ce848d2cbc3ec69f0ca"
Oct 30 22:57:02.843751 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:02.843607    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2b876b0153e506aff6408c123693f5d45dbc5fcb04e3ce848d2cbc3ec69f0ca"} err="failed to get container status \"d2b876b0153e506aff6408c123693f5d45dbc5fcb04e3ce848d2cbc3ec69f0ca\": rpc error: code = NotFound desc = could not find container \"d2b876b0153e506aff6408c123693f5d45dbc5fcb04e3ce848d2cbc3ec69f0ca\": container with ID starting with d2b876b0153e506aff6408c123693f5d45dbc5fcb04e3ce848d2cbc3ec69f0ca not found: ID does not exist"
Oct 30 22:57:02.874457 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:02.874419    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg"]
Oct 30 22:57:02.878009 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:02.877977    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:02.888039 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:02.888004    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c511a3-547c7f7b86-9dlzg"]
Oct 30 22:57:03.048046 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:03.048009    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:03.233567 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:03.233462    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:03.569191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:03.569156    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:04.759713 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:04.759683    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf"
Oct 30 22:57:04.778563 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:04.778524    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdd7407b-f54b-4e2a-8215-1b1a740b1d69" path="/var/lib/kubelet/pods/bdd7407b-f54b-4e2a-8215-1b1a740b1d69/volumes"
Oct 30 22:57:05.421058 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:05.421017    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:06.164815 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.164774    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"]
Oct 30 22:57:06.166027 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.166000    4889 cpu_manager.go:414] "containerMap: removing container" podUID="bdd7407b-f54b-4e2a-8215-1b1a740b1d69" containerName="maas-api"
Oct 30 22:57:06.166027 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.166026    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd7407b-f54b-4e2a-8215-1b1a740b1d69" containerName="maas-api"
Oct 30 22:57:06.166285 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.166211    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd7407b-f54b-4e2a-8215-1b1a740b1d69" containerName="maas-api"
Oct 30 22:57:06.172488 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.172459    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"
Oct 30 22:57:06.190809 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.190775    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"]
Oct 30 22:57:06.198055 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.198018    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-510f80\"/\"kube-root-ca.crt\""
Oct 30 22:57:06.198432 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.198296    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-510f80\"/\"openshift-service-ca.crt\""
Oct 30 22:57:06.202538 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.202509    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-510f80\"/\"rate-test-model-510f80-kserve-self-signed-certs\""
Oct 30 22:57:06.202538 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.202521    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-510f80\"/\"default-dockercfg-vlhjl\""
Oct 30 22:57:06.255660 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.255627    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-home\") pod \"rate-test-model-510f80-kserve-c948fb597-bnk7r\" (UID: \"df0f924c-9c38-4e79-afee-85e96d72af9a\") " pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"
Oct 30 22:57:06.255902 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.255761    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mj5r\" (UniqueName: \"kubernetes.io/projected/df0f924c-9c38-4e79-afee-85e96d72af9a-kube-api-access-6mj5r\") pod \"rate-test-model-510f80-kserve-c948fb597-bnk7r\" (UID: \"df0f924c-9c38-4e79-afee-85e96d72af9a\") " pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"
Oct 30 22:57:06.255902 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.255868    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-model-cache\") pod \"rate-test-model-510f80-kserve-c948fb597-bnk7r\" (UID: \"df0f924c-9c38-4e79-afee-85e96d72af9a\") " pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"
Oct 30 22:57:06.256005 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.255926    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df0f924c-9c38-4e79-afee-85e96d72af9a-tls-certs\") pod \"rate-test-model-510f80-kserve-c948fb597-bnk7r\" (UID: \"df0f924c-9c38-4e79-afee-85e96d72af9a\") " pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"
Oct 30 22:57:06.256060 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.256025    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-dshm\") pod \"rate-test-model-510f80-kserve-c948fb597-bnk7r\" (UID: \"df0f924c-9c38-4e79-afee-85e96d72af9a\") " pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"
Oct 30 22:57:06.256174 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.256151    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-tmp-dir\") pod \"rate-test-model-510f80-kserve-c948fb597-bnk7r\" (UID: \"df0f924c-9c38-4e79-afee-85e96d72af9a\") " pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"
Oct 30 22:57:06.273180 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.273148    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:06.357116 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.357081    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-model-cache\") pod \"rate-test-model-510f80-kserve-c948fb597-bnk7r\" (UID: \"df0f924c-9c38-4e79-afee-85e96d72af9a\") " pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"
Oct 30 22:57:06.357342 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.357128    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df0f924c-9c38-4e79-afee-85e96d72af9a-tls-certs\") pod \"rate-test-model-510f80-kserve-c948fb597-bnk7r\" (UID: \"df0f924c-9c38-4e79-afee-85e96d72af9a\") " pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"
Oct 30 22:57:06.357342 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.357215    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-dshm\") pod \"rate-test-model-510f80-kserve-c948fb597-bnk7r\" (UID: \"df0f924c-9c38-4e79-afee-85e96d72af9a\") " pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"
Oct 30 22:57:06.357342 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.357303    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-tmp-dir\") pod \"rate-test-model-510f80-kserve-c948fb597-bnk7r\" (UID: \"df0f924c-9c38-4e79-afee-85e96d72af9a\") " pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"
Oct 30 22:57:06.357566 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.357385    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-home\") pod \"rate-test-model-510f80-kserve-c948fb597-bnk7r\" (UID: \"df0f924c-9c38-4e79-afee-85e96d72af9a\") " pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"
Oct 30 22:57:06.357566 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.357452    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mj5r\" (UniqueName: \"kubernetes.io/projected/df0f924c-9c38-4e79-afee-85e96d72af9a-kube-api-access-6mj5r\") pod \"rate-test-model-510f80-kserve-c948fb597-bnk7r\" (UID: \"df0f924c-9c38-4e79-afee-85e96d72af9a\") " pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"
Oct 30 22:57:06.357654 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.357626    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-model-cache\") pod \"rate-test-model-510f80-kserve-c948fb597-bnk7r\" (UID: \"df0f924c-9c38-4e79-afee-85e96d72af9a\") " pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"
Oct 30 22:57:06.357693 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.357669    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-tmp-dir\") pod \"rate-test-model-510f80-kserve-c948fb597-bnk7r\" (UID: \"df0f924c-9c38-4e79-afee-85e96d72af9a\") " pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"
Oct 30 22:57:06.357734 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.357690    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-home\") pod \"rate-test-model-510f80-kserve-c948fb597-bnk7r\" (UID: \"df0f924c-9c38-4e79-afee-85e96d72af9a\") " pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"
Oct 30 22:57:06.359696 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.359672    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-dshm\") pod \"rate-test-model-510f80-kserve-c948fb597-bnk7r\" (UID: \"df0f924c-9c38-4e79-afee-85e96d72af9a\") " pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"
Oct 30 22:57:06.360008 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.359986    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df0f924c-9c38-4e79-afee-85e96d72af9a-tls-certs\") pod \"rate-test-model-510f80-kserve-c948fb597-bnk7r\" (UID: \"df0f924c-9c38-4e79-afee-85e96d72af9a\") " pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"
Oct 30 22:57:06.378825 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.378795    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mj5r\" (UniqueName: \"kubernetes.io/projected/df0f924c-9c38-4e79-afee-85e96d72af9a-kube-api-access-6mj5r\") pod \"rate-test-model-510f80-kserve-c948fb597-bnk7r\" (UID: \"df0f924c-9c38-4e79-afee-85e96d72af9a\") " pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"
Oct 30 22:57:06.493026 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.492994    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"
Oct 30 22:57:06.813711 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:57:06.813676    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf0f924c_9c38_4e79_afee_85e96d72af9a.slice/crio-9db82784faa3b5655294add357d0c602efd877de471af1a610e4762887d810ac WatchSource:0}: Error finding container 9db82784faa3b5655294add357d0c602efd877de471af1a610e4762887d810ac: Status 404 returned error can't find the container with id 9db82784faa3b5655294add357d0c602efd877de471af1a610e4762887d810ac
Oct 30 22:57:06.815123 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.815096    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"]
Oct 30 22:57:06.850150 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:06.850112    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r" event={"ID":"df0f924c-9c38-4e79-afee-85e96d72af9a","Type":"ContainerStarted","Data":"9db82784faa3b5655294add357d0c602efd877de471af1a610e4762887d810ac"}
Oct 30 22:57:07.204188 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:07.204092    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:07.863794 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:07.863745    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r" event={"ID":"df0f924c-9c38-4e79-afee-85e96d72af9a","Type":"ContainerStarted","Data":"1acf8c4182a28eae86167db482bc123cd0b9de27e3b4aa4b6e07f5faddda1233"}
Oct 30 22:57:07.913155 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:07.913087    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r" podStartSLOduration=1.913070689 podStartE2EDuration="1.913070689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:57:07.910909991 +0000 UTC m=+3013.267276307" watchObservedRunningTime="2026-10-30 22:57:07.913070689 +0000 UTC m=+3013.269437003"
Oct 30 22:57:08.493471 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:08.493430    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"
Oct 30 22:57:08.952876 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:08.952791    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:10.305122 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:57:10.305075    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:57:10.546309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:10.546275    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:10.720437 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:10.720340    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:10.831891 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:10.831856    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"]
Oct 30 22:57:10.832105 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:10.832083    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr" podUID="5a707459-e53d-486b-a1e5-0456ab08570c" containerName="istio-proxy" containerID="cri-o://6affa35bb38bd78180c641c0599cf7c658d43284dd518ebf04357db101246737" gracePeriod=30
Oct 30 22:57:10.930939 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:10.930588    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:11.149035 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:11.148996    4889 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:57:11.149270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:11.149070    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr" podUID="5a707459-e53d-486b-a1e5-0456ab08570c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:57:12.855147 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:12.855109    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:15.128734 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:15.128687    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:16.145813 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.145770    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:57:16.217516 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.217480    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a707459-e53d-486b-a1e5-0456ab08570c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a707459-e53d-486b-a1e5-0456ab08570c-istiod-ca-cert\") pod \"5a707459-e53d-486b-a1e5-0456ab08570c\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") "
Oct 30 22:57:16.217741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.217563    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-credential-socket\") pod \"5a707459-e53d-486b-a1e5-0456ab08570c\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") "
Oct 30 22:57:16.217741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.217586    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a707459-e53d-486b-a1e5-0456ab08570c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a707459-e53d-486b-a1e5-0456ab08570c-istio-podinfo\") pod \"5a707459-e53d-486b-a1e5-0456ab08570c\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") "
Oct 30 22:57:16.217741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.217666    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-istio-data\") pod \"5a707459-e53d-486b-a1e5-0456ab08570c\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") "
Oct 30 22:57:16.217741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.217703    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-istio-envoy\") pod \"5a707459-e53d-486b-a1e5-0456ab08570c\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") "
Oct 30 22:57:16.218024 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.217911    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-credential-socket" pod "5a707459-e53d-486b-a1e5-0456ab08570c" (UID: "5a707459-e53d-486b-a1e5-0456ab08570c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:16.218024 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.217941    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a707459-e53d-486b-a1e5-0456ab08570c-istiod-ca-cert" pod "5a707459-e53d-486b-a1e5-0456ab08570c" (UID: "5a707459-e53d-486b-a1e5-0456ab08570c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:57:16.218024 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.217994    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-workload-certs\") pod \"5a707459-e53d-486b-a1e5-0456ab08570c\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") "
Oct 30 22:57:16.218252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.218046    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a707459-e53d-486b-a1e5-0456ab08570c-istio-token\" (UniqueName: \"kubernetes.io/projected/5a707459-e53d-486b-a1e5-0456ab08570c-istio-token\") pod \"5a707459-e53d-486b-a1e5-0456ab08570c\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") "
Oct 30 22:57:16.218252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.218150    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-istio-data" pod "5a707459-e53d-486b-a1e5-0456ab08570c" (UID: "5a707459-e53d-486b-a1e5-0456ab08570c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:16.218252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.218176    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a707459-e53d-486b-a1e5-0456ab08570c-kube-api-access-xqmtc\" (UniqueName: \"kubernetes.io/projected/5a707459-e53d-486b-a1e5-0456ab08570c-kube-api-access-xqmtc\") pod \"5a707459-e53d-486b-a1e5-0456ab08570c\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") "
Oct 30 22:57:16.218408 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.218344    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-workload-certs" pod "5a707459-e53d-486b-a1e5-0456ab08570c" (UID: "5a707459-e53d-486b-a1e5-0456ab08570c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:16.218463 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.218434    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-workload-socket\") pod \"5a707459-e53d-486b-a1e5-0456ab08570c\" (UID: \"5a707459-e53d-486b-a1e5-0456ab08570c\") "
Oct 30 22:57:16.218804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.218777    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-workload-socket" pod "5a707459-e53d-486b-a1e5-0456ab08570c" (UID: "5a707459-e53d-486b-a1e5-0456ab08570c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:16.219321 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.219295    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:57:16.219479 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.219462    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a707459-e53d-486b-a1e5-0456ab08570c-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:57:16.219587 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.219573    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:57:16.219691 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.219678    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:57:16.219786 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.219775    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:57:16.224036 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.220951    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-istio-envoy" pod "5a707459-e53d-486b-a1e5-0456ab08570c" (UID: "5a707459-e53d-486b-a1e5-0456ab08570c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:16.224036 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.221076    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a707459-e53d-486b-a1e5-0456ab08570c-istio-token" pod "5a707459-e53d-486b-a1e5-0456ab08570c" (UID: "5a707459-e53d-486b-a1e5-0456ab08570c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:16.224036 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.221912    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a707459-e53d-486b-a1e5-0456ab08570c-istio-podinfo" pod "5a707459-e53d-486b-a1e5-0456ab08570c" (UID: "5a707459-e53d-486b-a1e5-0456ab08570c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:57:16.224036 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.222475    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a707459-e53d-486b-a1e5-0456ab08570c-kube-api-access-xqmtc" pod "5a707459-e53d-486b-a1e5-0456ab08570c" (UID: "5a707459-e53d-486b-a1e5-0456ab08570c"). InnerVolumeSpecName "kube-api-access-xqmtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:16.321481 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.321443    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a707459-e53d-486b-a1e5-0456ab08570c-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:57:16.321481 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.321482    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a707459-e53d-486b-a1e5-0456ab08570c-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:57:16.321481 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.321491    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a707459-e53d-486b-a1e5-0456ab08570c-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:57:16.321746 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.321500    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqmtc\" (UniqueName: \"kubernetes.io/projected/5a707459-e53d-486b-a1e5-0456ab08570c-kube-api-access-xqmtc\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:57:16.493380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.493343    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"
Oct 30 22:57:16.506481 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.506447    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"
Oct 30 22:57:16.780586 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.780506    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w"]
Oct 30 22:57:16.781797 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.781770    4889 cpu_manager.go:414] "containerMap: removing container" podUID="5a707459-e53d-486b-a1e5-0456ab08570c" containerName="istio-proxy"
Oct 30 22:57:16.781855 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.781803    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a707459-e53d-486b-a1e5-0456ab08570c" containerName="istio-proxy"
Oct 30 22:57:16.782038 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.782024    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a707459-e53d-486b-a1e5-0456ab08570c" containerName="istio-proxy"
Oct 30 22:57:16.789394 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.788363    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w"
Oct 30 22:57:16.798331 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.795993    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w"]
Oct 30 22:57:16.813433 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.813396    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f88979-serving-cert\""
Oct 30 22:57:16.829104 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.829065    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/457f9b6e-69a8-49d4-b65a-5afe56213f23-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w\" (UID: \"457f9b6e-69a8-49d4-b65a-5afe56213f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w"
Oct 30 22:57:16.829415 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.829380    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdtr5\" (UniqueName: \"kubernetes.io/projected/457f9b6e-69a8-49d4-b65a-5afe56213f23-kube-api-access-xdtr5\") pod \"maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w\" (UID: \"457f9b6e-69a8-49d4-b65a-5afe56213f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w"
Oct 30 22:57:16.829530 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.829511    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/457f9b6e-69a8-49d4-b65a-5afe56213f23-metrics-certs\") pod \"maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w\" (UID: \"457f9b6e-69a8-49d4-b65a-5afe56213f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w"
Oct 30 22:57:16.829605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.829561    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/457f9b6e-69a8-49d4-b65a-5afe56213f23-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w\" (UID: \"457f9b6e-69a8-49d4-b65a-5afe56213f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w"
Oct 30 22:57:16.905995 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.905960    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:16.930789 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.930746    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdtr5\" (UniqueName: \"kubernetes.io/projected/457f9b6e-69a8-49d4-b65a-5afe56213f23-kube-api-access-xdtr5\") pod \"maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w\" (UID: \"457f9b6e-69a8-49d4-b65a-5afe56213f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w"
Oct 30 22:57:16.930980 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.930844    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/457f9b6e-69a8-49d4-b65a-5afe56213f23-metrics-certs\") pod \"maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w\" (UID: \"457f9b6e-69a8-49d4-b65a-5afe56213f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w"
Oct 30 22:57:16.930980 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.930883    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/457f9b6e-69a8-49d4-b65a-5afe56213f23-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w\" (UID: \"457f9b6e-69a8-49d4-b65a-5afe56213f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w"
Oct 30 22:57:16.930980 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.930941    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/457f9b6e-69a8-49d4-b65a-5afe56213f23-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w\" (UID: \"457f9b6e-69a8-49d4-b65a-5afe56213f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w"
Oct 30 22:57:16.931904 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.931836    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/457f9b6e-69a8-49d4-b65a-5afe56213f23-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w\" (UID: \"457f9b6e-69a8-49d4-b65a-5afe56213f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w"
Oct 30 22:57:16.934197 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.934164    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/457f9b6e-69a8-49d4-b65a-5afe56213f23-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w\" (UID: \"457f9b6e-69a8-49d4-b65a-5afe56213f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w"
Oct 30 22:57:16.952305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.948403    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/457f9b6e-69a8-49d4-b65a-5afe56213f23-metrics-certs\") pod \"maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w\" (UID: \"457f9b6e-69a8-49d4-b65a-5afe56213f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w"
Oct 30 22:57:16.954936 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.954294    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdtr5\" (UniqueName: \"kubernetes.io/projected/457f9b6e-69a8-49d4-b65a-5afe56213f23-kube-api-access-xdtr5\") pod \"maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w\" (UID: \"457f9b6e-69a8-49d4-b65a-5afe56213f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w"
Oct 30 22:57:16.963621 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.963577    4889 generic.go:356] "Generic (PLEG): container finished" podID="5a707459-e53d-486b-a1e5-0456ab08570c" containerID="6affa35bb38bd78180c641c0599cf7c658d43284dd518ebf04357db101246737" exitCode=0
Oct 30 22:57:16.966872 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.963987    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"
Oct 30 22:57:16.966872 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.964209    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr" event={"ID":"5a707459-e53d-486b-a1e5-0456ab08570c","Type":"ContainerDied","Data":"6affa35bb38bd78180c641c0599cf7c658d43284dd518ebf04357db101246737"}
Oct 30 22:57:16.966872 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.964850    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr" event={"ID":"5a707459-e53d-486b-a1e5-0456ab08570c","Type":"ContainerDied","Data":"65714b706e2728e74b35ec51b428bf1594b8bdc800406551ef8e76c706665cf5"}
Oct 30 22:57:16.966872 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.964903    4889 scope.go:122] "RemoveContainer" containerID="6affa35bb38bd78180c641c0599cf7c658d43284dd518ebf04357db101246737"
Oct 30 22:57:16.979638 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.979607    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"
Oct 30 22:57:16.992634 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:16.992579    4889 scope.go:122] "RemoveContainer" containerID="981bd4e0cb5bd05d6b76026ccab69712b1d990a628fddfae3a194e1f3c7d624c"
Oct 30 22:57:17.005416 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:17.005392    4889 scope.go:122] "RemoveContainer" containerID="6affa35bb38bd78180c641c0599cf7c658d43284dd518ebf04357db101246737"
Oct 30 22:57:17.005752 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:57:17.005733    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6affa35bb38bd78180c641c0599cf7c658d43284dd518ebf04357db101246737\": container with ID starting with 6affa35bb38bd78180c641c0599cf7c658d43284dd518ebf04357db101246737 not found: ID does not exist" containerID="6affa35bb38bd78180c641c0599cf7c658d43284dd518ebf04357db101246737"
Oct 30 22:57:17.005847 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:17.005764    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6affa35bb38bd78180c641c0599cf7c658d43284dd518ebf04357db101246737"} err="failed to get container status \"6affa35bb38bd78180c641c0599cf7c658d43284dd518ebf04357db101246737\": rpc error: code = NotFound desc = could not find container \"6affa35bb38bd78180c641c0599cf7c658d43284dd518ebf04357db101246737\": container with ID starting with 6affa35bb38bd78180c641c0599cf7c658d43284dd518ebf04357db101246737 not found: ID does not exist"
Oct 30 22:57:17.005847 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:17.005790    4889 scope.go:122] "RemoveContainer" containerID="981bd4e0cb5bd05d6b76026ccab69712b1d990a628fddfae3a194e1f3c7d624c"
Oct 30 22:57:17.006139 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:57:17.006119    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"981bd4e0cb5bd05d6b76026ccab69712b1d990a628fddfae3a194e1f3c7d624c\": container with ID starting with 981bd4e0cb5bd05d6b76026ccab69712b1d990a628fddfae3a194e1f3c7d624c not found: ID does not exist" containerID="981bd4e0cb5bd05d6b76026ccab69712b1d990a628fddfae3a194e1f3c7d624c"
Oct 30 22:57:17.006195 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:17.006145    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"981bd4e0cb5bd05d6b76026ccab69712b1d990a628fddfae3a194e1f3c7d624c"} err="failed to get container status \"981bd4e0cb5bd05d6b76026ccab69712b1d990a628fddfae3a194e1f3c7d624c\": rpc error: code = NotFound desc = could not find container \"981bd4e0cb5bd05d6b76026ccab69712b1d990a628fddfae3a194e1f3c7d624c\": container with ID starting with 981bd4e0cb5bd05d6b76026ccab69712b1d990a628fddfae3a194e1f3c7d624c not found: ID does not exist"
Oct 30 22:57:17.068925 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:17.068809    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"]
Oct 30 22:57:17.077786 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:17.077747    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c511a3-openshift-default-59bc6d8b7-9m4nr"]
Oct 30 22:57:17.097972 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:17.097931    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:17.119101 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:17.119059    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w"
Oct 30 22:57:17.305846 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:17.305811    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:17.435359 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:57:17.435322    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457f9b6e_69a8_49d4_b65a_5afe56213f23.slice/crio-462963f02d408952bfc782762c7226d2f35a2ee926424fea0734cd29f1862802 WatchSource:0}: Error finding container 462963f02d408952bfc782762c7226d2f35a2ee926424fea0734cd29f1862802: Status 404 returned error can't find the container with id 462963f02d408952bfc782762c7226d2f35a2ee926424fea0734cd29f1862802
Oct 30 22:57:17.435780 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:17.435753    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w"]
Oct 30 22:57:17.995414 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:17.995373    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w" event={"ID":"457f9b6e-69a8-49d4-b65a-5afe56213f23","Type":"ContainerStarted","Data":"90653e59849e1d176fa765386ad14815a6eed82e88789efcc79d56129a1a3c02"}
Oct 30 22:57:17.995640 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:17.995440    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w" event={"ID":"457f9b6e-69a8-49d4-b65a-5afe56213f23","Type":"ContainerStarted","Data":"462963f02d408952bfc782762c7226d2f35a2ee926424fea0734cd29f1862802"}
Oct 30 22:57:17.995640 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:17.995470    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w"
Oct 30 22:57:18.039436 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:18.039374    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w" podStartSLOduration=1.616589941 podStartE2EDuration="2.039358297s" totalImagesPullingTime="422.768356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:57:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:57:18.037651165 +0000 UTC m=+3023.394017479" watchObservedRunningTime="2026-10-30 22:57:18.039358297 +0000 UTC m=+3023.395724676"
Oct 30 22:57:18.764941 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:18.764898    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:18.778383 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:18.778318    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a707459-e53d-486b-a1e5-0456ab08570c" path="/var/lib/kubelet/pods/5a707459-e53d-486b-a1e5-0456ab08570c/volumes"
Oct 30 22:57:20.529391 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:20.529349    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:22.067907 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:22.067868    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:22.282027 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:22.281986    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:22.544254 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:22.537381    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:24.009583 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:24.009548    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w"
Oct 30 22:57:24.259145 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:24.259082    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:26.049506 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:26.049454    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:27.979155 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:27.979110    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:29.718620 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:29.718581    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:31.402755 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:31.402714    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:33.251709 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:33.251671    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:35.008569 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:35.008529    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:36.688363 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:36.688325    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:38.457662 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:38.457621    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:40.144108 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:40.144069    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:40.303975 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:57:40.303935    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:57:41.928885 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:41.928845    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:43.724657 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:43.724601    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:45.429602 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:45.429516    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:47.311848 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:47.311809    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:49.169276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:49.169203    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:50.289035 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:50.288998    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"]
Oct 30 22:57:50.390827 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:50.390789    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:50.688054 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:50.687967    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:50.790984 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:50.790940    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8"]
Oct 30 22:57:51.051191 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:51.051155    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:51.224000 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:51.223962    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:51.465311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:51.455202    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:53.135626 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:53.135586    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:54.817633 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:54.817597    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:55.791459 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:55.791407    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8" podUID="512c5c51-57a4-48d3-b1c3-e1fdb7bbd483" containerName="payload-processing" containerID="cri-o://0a661806e131496ac6322cc2a6413738de2e0aec3c285c1bb83eacd28ce0bfe5" gracePeriod=25
Oct 30 22:57:56.094435 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:56.094400    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8"
Oct 30 22:57:56.170688 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:56.170652    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/512c5c51-57a4-48d3-b1c3-e1fdb7bbd483-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/512c5c51-57a4-48d3-b1c3-e1fdb7bbd483-plugins-config-volume\") pod \"512c5c51-57a4-48d3-b1c3-e1fdb7bbd483\" (UID: \"512c5c51-57a4-48d3-b1c3-e1fdb7bbd483\") "
Oct 30 22:57:56.170995 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:56.170855    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/512c5c51-57a4-48d3-b1c3-e1fdb7bbd483-kube-api-access-2hm6n\" (UniqueName: \"kubernetes.io/projected/512c5c51-57a4-48d3-b1c3-e1fdb7bbd483-kube-api-access-2hm6n\") pod \"512c5c51-57a4-48d3-b1c3-e1fdb7bbd483\" (UID: \"512c5c51-57a4-48d3-b1c3-e1fdb7bbd483\") "
Oct 30 22:57:56.171255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:56.171188    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/512c5c51-57a4-48d3-b1c3-e1fdb7bbd483-plugins-config-volume" pod "512c5c51-57a4-48d3-b1c3-e1fdb7bbd483" (UID: "512c5c51-57a4-48d3-b1c3-e1fdb7bbd483"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:57:56.175142 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:56.175114    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/512c5c51-57a4-48d3-b1c3-e1fdb7bbd483-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:57:56.175333 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:56.175137    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/512c5c51-57a4-48d3-b1c3-e1fdb7bbd483-kube-api-access-2hm6n" pod "512c5c51-57a4-48d3-b1c3-e1fdb7bbd483" (UID: "512c5c51-57a4-48d3-b1c3-e1fdb7bbd483"). InnerVolumeSpecName "kube-api-access-2hm6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:56.277094 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:56.277047    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hm6n\" (UniqueName: \"kubernetes.io/projected/512c5c51-57a4-48d3-b1c3-e1fdb7bbd483-kube-api-access-2hm6n\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:57:56.319112 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:56.319013    4889 generic.go:356] "Generic (PLEG): container finished" podID="512c5c51-57a4-48d3-b1c3-e1fdb7bbd483" containerID="0a661806e131496ac6322cc2a6413738de2e0aec3c285c1bb83eacd28ce0bfe5" exitCode=0
Oct 30 22:57:56.319112 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:56.319073    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8"
Oct 30 22:57:56.319431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:56.319061    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8" event={"ID":"512c5c51-57a4-48d3-b1c3-e1fdb7bbd483","Type":"ContainerDied","Data":"0a661806e131496ac6322cc2a6413738de2e0aec3c285c1bb83eacd28ce0bfe5"}
Oct 30 22:57:56.319431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:56.319239    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8" event={"ID":"512c5c51-57a4-48d3-b1c3-e1fdb7bbd483","Type":"ContainerDied","Data":"61630b4b627b6ab4dea053cba3cd892f0099720e8f2a09fb99b8cafc8c13e921"}
Oct 30 22:57:56.319431 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:56.319276    4889 scope.go:122] "RemoveContainer" containerID="0a661806e131496ac6322cc2a6413738de2e0aec3c285c1bb83eacd28ce0bfe5"
Oct 30 22:57:56.333039 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:56.333012    4889 scope.go:122] "RemoveContainer" containerID="0a661806e131496ac6322cc2a6413738de2e0aec3c285c1bb83eacd28ce0bfe5"
Oct 30 22:57:56.333415 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:57:56.333394    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a661806e131496ac6322cc2a6413738de2e0aec3c285c1bb83eacd28ce0bfe5\": container with ID starting with 0a661806e131496ac6322cc2a6413738de2e0aec3c285c1bb83eacd28ce0bfe5 not found: ID does not exist" containerID="0a661806e131496ac6322cc2a6413738de2e0aec3c285c1bb83eacd28ce0bfe5"
Oct 30 22:57:56.333490 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:56.333427    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a661806e131496ac6322cc2a6413738de2e0aec3c285c1bb83eacd28ce0bfe5"} err="failed to get container status \"0a661806e131496ac6322cc2a6413738de2e0aec3c285c1bb83eacd28ce0bfe5\": rpc error: code = NotFound desc = could not find container \"0a661806e131496ac6322cc2a6413738de2e0aec3c285c1bb83eacd28ce0bfe5\": container with ID starting with 0a661806e131496ac6322cc2a6413738de2e0aec3c285c1bb83eacd28ce0bfe5 not found: ID does not exist"
Oct 30 22:57:56.378115 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:56.378066    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8"]
Oct 30 22:57:56.381829 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:56.381799    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0a807-6968cf6bcc-xbgf8"]
Oct 30 22:57:56.712012 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:56.711904    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:56.778346 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:56.778310    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="512c5c51-57a4-48d3-b1c3-e1fdb7bbd483" path="/var/lib/kubelet/pods/512c5c51-57a4-48d3-b1c3-e1fdb7bbd483/volumes"
Oct 30 22:57:58.447616 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:57:58.447562    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:00.723681 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:00.723639    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:02.527933 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:02.527890    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:04.414314 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:04.410903    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:04.640993 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:04.640956    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:05.313866 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:05.313805    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb" podUID="2f8596b1-c420-46cd-85cd-6b68dffff980" containerName="main" containerID="cri-o://ffb19f8b6a11749f5f9fde8f2b92388f71aae2b36e28d152a2cb91678991de00" gracePeriod=45
Oct 30 22:58:05.468617 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:05.468560    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb" podUID="2f8596b1-c420-46cd-85cd-6b68dffff980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.218:8000/ready\": dial tcp 10.132.0.218:8000: connect: connection refused"
Oct 30 22:58:05.610940 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:05.610912    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"
Oct 30 22:58:05.725736 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:05.725689    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-home\" (UniqueName: \"kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-home\") pod \"2f8596b1-c420-46cd-85cd-6b68dffff980\" (UID: \"2f8596b1-c420-46cd-85cd-6b68dffff980\") "
Oct 30 22:58:05.725958 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:05.725869    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-tmp-dir\") pod \"2f8596b1-c420-46cd-85cd-6b68dffff980\" (UID: \"2f8596b1-c420-46cd-85cd-6b68dffff980\") "
Oct 30 22:58:05.726044 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:05.726003    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-home" pod "2f8596b1-c420-46cd-85cd-6b68dffff980" (UID: "2f8596b1-c420-46cd-85cd-6b68dffff980"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:05.726113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:05.726076    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f8596b1-c420-46cd-85cd-6b68dffff980-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f8596b1-c420-46cd-85cd-6b68dffff980-tls-certs\") pod \"2f8596b1-c420-46cd-85cd-6b68dffff980\" (UID: \"2f8596b1-c420-46cd-85cd-6b68dffff980\") "
Oct 30 22:58:05.726175 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:05.726131    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-model-cache\") pod \"2f8596b1-c420-46cd-85cd-6b68dffff980\" (UID: \"2f8596b1-c420-46cd-85cd-6b68dffff980\") "
Oct 30 22:58:05.726297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:05.726175    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-tmp-dir" pod "2f8596b1-c420-46cd-85cd-6b68dffff980" (UID: "2f8596b1-c420-46cd-85cd-6b68dffff980"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:05.726297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:05.726271    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-dshm\") pod \"2f8596b1-c420-46cd-85cd-6b68dffff980\" (UID: \"2f8596b1-c420-46cd-85cd-6b68dffff980\") "
Oct 30 22:58:05.726437 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:05.726316    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f8596b1-c420-46cd-85cd-6b68dffff980-kube-api-access-2lphb\" (UniqueName: \"kubernetes.io/projected/2f8596b1-c420-46cd-85cd-6b68dffff980-kube-api-access-2lphb\") pod \"2f8596b1-c420-46cd-85cd-6b68dffff980\" (UID: \"2f8596b1-c420-46cd-85cd-6b68dffff980\") "
Oct 30 22:58:05.726662 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:05.726631    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-model-cache" pod "2f8596b1-c420-46cd-85cd-6b68dffff980" (UID: "2f8596b1-c420-46cd-85cd-6b68dffff980"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:05.727328 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:05.727304    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:05.727429 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:05.727359    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:05.727429 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:05.727380    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:05.728878 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:05.728855    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f8596b1-c420-46cd-85cd-6b68dffff980-tls-certs" pod "2f8596b1-c420-46cd-85cd-6b68dffff980" (UID: "2f8596b1-c420-46cd-85cd-6b68dffff980"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:58:05.729003 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:05.728916    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-dshm" pod "2f8596b1-c420-46cd-85cd-6b68dffff980" (UID: "2f8596b1-c420-46cd-85cd-6b68dffff980"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:05.729107 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:05.729084    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8596b1-c420-46cd-85cd-6b68dffff980-kube-api-access-2lphb" pod "2f8596b1-c420-46cd-85cd-6b68dffff980" (UID: "2f8596b1-c420-46cd-85cd-6b68dffff980"). InnerVolumeSpecName "kube-api-access-2lphb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:58:05.828793 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:05.828696    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f8596b1-c420-46cd-85cd-6b68dffff980-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:05.828793 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:05.828730    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f8596b1-c420-46cd-85cd-6b68dffff980-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:05.828793 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:05.828741    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lphb\" (UniqueName: \"kubernetes.io/projected/2f8596b1-c420-46cd-85cd-6b68dffff980-kube-api-access-2lphb\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:05.883013 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:05.882977    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk"]
Oct 30 22:58:05.883892 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:05.883209    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk" podUID="0997827b-80e4-4077-a22f-3d2026492158" containerName="maas-api" containerID="cri-o://ce5d16e10772ddd7277ad2ad696b54f4f282bd3e724731b834d5ac486bbab865" gracePeriod=30
Oct 30 22:58:05.997459 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:05.997423    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:06.171323 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.169277    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk"
Oct 30 22:58:06.336780 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.336750    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0997827b-80e4-4077-a22f-3d2026492158-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0997827b-80e4-4077-a22f-3d2026492158-openshift-service-ca\") pod \"0997827b-80e4-4077-a22f-3d2026492158\" (UID: \"0997827b-80e4-4077-a22f-3d2026492158\") "
Oct 30 22:58:06.337014 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.336995    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0997827b-80e4-4077-a22f-3d2026492158-kube-api-access-xmtf2\" (UniqueName: \"kubernetes.io/projected/0997827b-80e4-4077-a22f-3d2026492158-kube-api-access-xmtf2\") pod \"0997827b-80e4-4077-a22f-3d2026492158\" (UID: \"0997827b-80e4-4077-a22f-3d2026492158\") "
Oct 30 22:58:06.337106 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.337065    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0997827b-80e4-4077-a22f-3d2026492158-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0997827b-80e4-4077-a22f-3d2026492158-metrics-certs\") pod \"0997827b-80e4-4077-a22f-3d2026492158\" (UID: \"0997827b-80e4-4077-a22f-3d2026492158\") "
Oct 30 22:58:06.337173 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.337112    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0997827b-80e4-4077-a22f-3d2026492158-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0997827b-80e4-4077-a22f-3d2026492158-maas-api-tls\") pod \"0997827b-80e4-4077-a22f-3d2026492158\" (UID: \"0997827b-80e4-4077-a22f-3d2026492158\") "
Oct 30 22:58:06.337255 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.337186    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0997827b-80e4-4077-a22f-3d2026492158-openshift-service-ca" pod "0997827b-80e4-4077-a22f-3d2026492158" (UID: "0997827b-80e4-4077-a22f-3d2026492158"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:58:06.337594 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.337571    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0997827b-80e4-4077-a22f-3d2026492158-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:06.339647 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.339614    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0997827b-80e4-4077-a22f-3d2026492158-maas-api-tls" pod "0997827b-80e4-4077-a22f-3d2026492158" (UID: "0997827b-80e4-4077-a22f-3d2026492158"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:58:06.339939 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.339916    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0997827b-80e4-4077-a22f-3d2026492158-metrics-certs" pod "0997827b-80e4-4077-a22f-3d2026492158" (UID: "0997827b-80e4-4077-a22f-3d2026492158"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:58:06.340020 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.339983    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0997827b-80e4-4077-a22f-3d2026492158-kube-api-access-xmtf2" pod "0997827b-80e4-4077-a22f-3d2026492158" (UID: "0997827b-80e4-4077-a22f-3d2026492158"). InnerVolumeSpecName "kube-api-access-xmtf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:58:06.422410 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.422307    4889 generic.go:356] "Generic (PLEG): container finished" podID="0997827b-80e4-4077-a22f-3d2026492158" containerID="ce5d16e10772ddd7277ad2ad696b54f4f282bd3e724731b834d5ac486bbab865" exitCode=0
Oct 30 22:58:06.422593 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.422537    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk"
Oct 30 22:58:06.423579 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.423548    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk" event={"ID":"0997827b-80e4-4077-a22f-3d2026492158","Type":"ContainerDied","Data":"ce5d16e10772ddd7277ad2ad696b54f4f282bd3e724731b834d5ac486bbab865"}
Oct 30 22:58:06.423854 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.423816    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk" event={"ID":"0997827b-80e4-4077-a22f-3d2026492158","Type":"ContainerDied","Data":"d1fd45c884bc97ac128b61ab45bf119ec4af3a9392ef26b145286b77796d5549"}
Oct 30 22:58:06.424028 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.424011    4889 scope.go:122] "RemoveContainer" containerID="ce5d16e10772ddd7277ad2ad696b54f4f282bd3e724731b834d5ac486bbab865"
Oct 30 22:58:06.426794 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.425997    4889 generic.go:356] "Generic (PLEG): container finished" podID="2f8596b1-c420-46cd-85cd-6b68dffff980" containerID="ffb19f8b6a11749f5f9fde8f2b92388f71aae2b36e28d152a2cb91678991de00" exitCode=0
Oct 30 22:58:06.426794 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.426276    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb" event={"ID":"2f8596b1-c420-46cd-85cd-6b68dffff980","Type":"ContainerDied","Data":"ffb19f8b6a11749f5f9fde8f2b92388f71aae2b36e28d152a2cb91678991de00"}
Oct 30 22:58:06.426794 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.426310    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb" event={"ID":"2f8596b1-c420-46cd-85cd-6b68dffff980","Type":"ContainerDied","Data":"6f3962d9968201991e47034324552a427a6cdda0b3870a397a249bfd155dad79"}
Oct 30 22:58:06.426794 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.426378    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"
Oct 30 22:58:06.439846 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.438993    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmtf2\" (UniqueName: \"kubernetes.io/projected/0997827b-80e4-4077-a22f-3d2026492158-kube-api-access-xmtf2\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:06.439846 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.439023    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0997827b-80e4-4077-a22f-3d2026492158-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:06.439846 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.439052    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0997827b-80e4-4077-a22f-3d2026492158-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:06.454426 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.454403    4889 scope.go:122] "RemoveContainer" containerID="ce5d16e10772ddd7277ad2ad696b54f4f282bd3e724731b834d5ac486bbab865"
Oct 30 22:58:06.455166 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:58:06.454960    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce5d16e10772ddd7277ad2ad696b54f4f282bd3e724731b834d5ac486bbab865\": container with ID starting with ce5d16e10772ddd7277ad2ad696b54f4f282bd3e724731b834d5ac486bbab865 not found: ID does not exist" containerID="ce5d16e10772ddd7277ad2ad696b54f4f282bd3e724731b834d5ac486bbab865"
Oct 30 22:58:06.455166 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.455002    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce5d16e10772ddd7277ad2ad696b54f4f282bd3e724731b834d5ac486bbab865"} err="failed to get container status \"ce5d16e10772ddd7277ad2ad696b54f4f282bd3e724731b834d5ac486bbab865\": rpc error: code = NotFound desc = could not find container \"ce5d16e10772ddd7277ad2ad696b54f4f282bd3e724731b834d5ac486bbab865\": container with ID starting with ce5d16e10772ddd7277ad2ad696b54f4f282bd3e724731b834d5ac486bbab865 not found: ID does not exist"
Oct 30 22:58:06.455166 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.455028    4889 scope.go:122] "RemoveContainer" containerID="ffb19f8b6a11749f5f9fde8f2b92388f71aae2b36e28d152a2cb91678991de00"
Oct 30 22:58:06.471497 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.471469    4889 scope.go:122] "RemoveContainer" containerID="ffb19f8b6a11749f5f9fde8f2b92388f71aae2b36e28d152a2cb91678991de00"
Oct 30 22:58:06.471874 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:58:06.471846    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffb19f8b6a11749f5f9fde8f2b92388f71aae2b36e28d152a2cb91678991de00\": container with ID starting with ffb19f8b6a11749f5f9fde8f2b92388f71aae2b36e28d152a2cb91678991de00 not found: ID does not exist" containerID="ffb19f8b6a11749f5f9fde8f2b92388f71aae2b36e28d152a2cb91678991de00"
Oct 30 22:58:06.471937 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.471885    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffb19f8b6a11749f5f9fde8f2b92388f71aae2b36e28d152a2cb91678991de00"} err="failed to get container status \"ffb19f8b6a11749f5f9fde8f2b92388f71aae2b36e28d152a2cb91678991de00\": rpc error: code = NotFound desc = could not find container \"ffb19f8b6a11749f5f9fde8f2b92388f71aae2b36e28d152a2cb91678991de00\": container with ID starting with ffb19f8b6a11749f5f9fde8f2b92388f71aae2b36e28d152a2cb91678991de00 not found: ID does not exist"
Oct 30 22:58:06.498658 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.498615    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk"]
Oct 30 22:58:06.504009 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.503974    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b03b33-7c4946547b-b4rmk"]
Oct 30 22:58:06.530656 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.530620    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:06.542271 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.542188    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"]
Oct 30 22:58:06.547489 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.547457    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-bed544/auth-test-model-bed544-kserve-659fbc956f-jmcdb"]
Oct 30 22:58:06.703016 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.702915    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:06.776267 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.776212    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0997827b-80e4-4077-a22f-3d2026492158" path="/var/lib/kubelet/pods/0997827b-80e4-4077-a22f-3d2026492158/volumes"
Oct 30 22:58:06.776818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:06.776796    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f8596b1-c420-46cd-85cd-6b68dffff980" path="/var/lib/kubelet/pods/2f8596b1-c420-46cd-85cd-6b68dffff980/volumes"
Oct 30 22:58:07.145832 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:07.145794    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:08.377775 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:08.377737    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:08.776050 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:08.776020    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh"]
Oct 30 22:58:08.796119 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:08.796082    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv"]
Oct 30 22:58:08.965620 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:08.965583    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:09.145764 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:09.145616    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:10.144217 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:10.144169    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:10.304999 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:58:10.304961    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:58:11.836970 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:11.836846    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:13.368673 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:13.368597    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:13.542634 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:13.542489    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:13.656363 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:13.656271    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"]
Oct 30 22:58:13.656562 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:13.656527    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd" podUID="efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9" containerName="istio-proxy" containerID="cri-o://b1fbe6a13a71825e5dee3039efea5f663115f26bfd42233ee2845ac6bdf56ed1" gracePeriod=30
Oct 30 22:58:13.738980 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:13.738939    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:13.781262 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:13.778079    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh" podUID="7b7bae51-8941-42f0-93a1-0a20637b3b75" containerName="payload-processing" containerID="cri-o://b1c9192cb7a4940d27bad722beffdb1eb911d51bd6f72f07d304100d0743eebb" gracePeriod=25
Oct 30 22:58:13.812381 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:13.812343    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv" podUID="e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9" containerName="payload-pre-processing" containerID="cri-o://3edc0df224aa5dbdf84319a03fbc3baa81fb64fcc9064e11c8064bf882f8fa51" gracePeriod=25
Oct 30 22:58:14.149592 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.149555    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh"
Oct 30 22:58:14.191010 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.190936    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv"
Oct 30 22:58:14.230118 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.230077    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:14.264006 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.263965    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b7bae51-8941-42f0-93a1-0a20637b3b75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b7bae51-8941-42f0-93a1-0a20637b3b75-plugins-config-volume\") pod \"7b7bae51-8941-42f0-93a1-0a20637b3b75\" (UID: \"7b7bae51-8941-42f0-93a1-0a20637b3b75\") "
Oct 30 22:58:14.264194 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.264171    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9-kube-api-access-jzwqv\" (UniqueName: \"kubernetes.io/projected/e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9-kube-api-access-jzwqv\") pod \"e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9\" (UID: \"e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9\") "
Oct 30 22:58:14.264292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.264275    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9-plugins-config-volume\") pod \"e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9\" (UID: \"e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9\") "
Oct 30 22:58:14.264475 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.264446    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b7bae51-8941-42f0-93a1-0a20637b3b75-plugins-config-volume" pod "7b7bae51-8941-42f0-93a1-0a20637b3b75" (UID: "7b7bae51-8941-42f0-93a1-0a20637b3b75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:58:14.264553 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.264531    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b7bae51-8941-42f0-93a1-0a20637b3b75-kube-api-access-t7cgc\" (UniqueName: \"kubernetes.io/projected/7b7bae51-8941-42f0-93a1-0a20637b3b75-kube-api-access-t7cgc\") pod \"7b7bae51-8941-42f0-93a1-0a20637b3b75\" (UID: \"7b7bae51-8941-42f0-93a1-0a20637b3b75\") "
Oct 30 22:58:14.265024 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.264996    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9-plugins-config-volume" pod "e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9" (UID: "e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:58:14.265190 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.265163    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b7bae51-8941-42f0-93a1-0a20637b3b75-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:14.265190 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.265193    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:14.267069 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.267040    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b7bae51-8941-42f0-93a1-0a20637b3b75-kube-api-access-t7cgc" pod "7b7bae51-8941-42f0-93a1-0a20637b3b75" (UID: "7b7bae51-8941-42f0-93a1-0a20637b3b75"). InnerVolumeSpecName "kube-api-access-t7cgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:58:14.267177 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.267135    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9-kube-api-access-jzwqv" pod "e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9" (UID: "e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9"). InnerVolumeSpecName "kube-api-access-jzwqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:58:14.366150 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.366098    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7cgc\" (UniqueName: \"kubernetes.io/projected/7b7bae51-8941-42f0-93a1-0a20637b3b75-kube-api-access-t7cgc\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:14.366150 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.366141    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzwqv\" (UniqueName: \"kubernetes.io/projected/e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9-kube-api-access-jzwqv\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:14.495358 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.495325    4889 generic.go:356] "Generic (PLEG): container finished" podID="e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9" containerID="3edc0df224aa5dbdf84319a03fbc3baa81fb64fcc9064e11c8064bf882f8fa51" exitCode=0
Oct 30 22:58:14.495794 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.495370    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv"
Oct 30 22:58:14.495794 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.495399    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv" event={"ID":"e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9","Type":"ContainerDied","Data":"3edc0df224aa5dbdf84319a03fbc3baa81fb64fcc9064e11c8064bf882f8fa51"}
Oct 30 22:58:14.495794 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.495461    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv" event={"ID":"e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9","Type":"ContainerDied","Data":"6a7c83a8111825db21e6d77838d2efd28d1d3513f838dd8d2c7018f1bed8944b"}
Oct 30 22:58:14.495794 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.495493    4889 scope.go:122] "RemoveContainer" containerID="3edc0df224aa5dbdf84319a03fbc3baa81fb64fcc9064e11c8064bf882f8fa51"
Oct 30 22:58:14.499747 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.499712    4889 generic.go:356] "Generic (PLEG): container finished" podID="7b7bae51-8941-42f0-93a1-0a20637b3b75" containerID="b1c9192cb7a4940d27bad722beffdb1eb911d51bd6f72f07d304100d0743eebb" exitCode=0
Oct 30 22:58:14.499869 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.499773    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh"
Oct 30 22:58:14.499869 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.499792    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh" event={"ID":"7b7bae51-8941-42f0-93a1-0a20637b3b75","Type":"ContainerDied","Data":"b1c9192cb7a4940d27bad722beffdb1eb911d51bd6f72f07d304100d0743eebb"}
Oct 30 22:58:14.499869 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.499857    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh" event={"ID":"7b7bae51-8941-42f0-93a1-0a20637b3b75","Type":"ContainerDied","Data":"997c7e1675ac8ddfc238c7605457067c1d124f91044bd69b6055e66a11f40820"}
Oct 30 22:58:14.512181 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.512150    4889 scope.go:122] "RemoveContainer" containerID="3edc0df224aa5dbdf84319a03fbc3baa81fb64fcc9064e11c8064bf882f8fa51"
Oct 30 22:58:14.512519 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:58:14.512493    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3edc0df224aa5dbdf84319a03fbc3baa81fb64fcc9064e11c8064bf882f8fa51\": container with ID starting with 3edc0df224aa5dbdf84319a03fbc3baa81fb64fcc9064e11c8064bf882f8fa51 not found: ID does not exist" containerID="3edc0df224aa5dbdf84319a03fbc3baa81fb64fcc9064e11c8064bf882f8fa51"
Oct 30 22:58:14.512601 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.512531    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3edc0df224aa5dbdf84319a03fbc3baa81fb64fcc9064e11c8064bf882f8fa51"} err="failed to get container status \"3edc0df224aa5dbdf84319a03fbc3baa81fb64fcc9064e11c8064bf882f8fa51\": rpc error: code = NotFound desc = could not find container \"3edc0df224aa5dbdf84319a03fbc3baa81fb64fcc9064e11c8064bf882f8fa51\": container with ID starting with 3edc0df224aa5dbdf84319a03fbc3baa81fb64fcc9064e11c8064bf882f8fa51 not found: ID does not exist"
Oct 30 22:58:14.512601 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.512550    4889 scope.go:122] "RemoveContainer" containerID="b1c9192cb7a4940d27bad722beffdb1eb911d51bd6f72f07d304100d0743eebb"
Oct 30 22:58:14.523630 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.523599    4889 scope.go:122] "RemoveContainer" containerID="b1c9192cb7a4940d27bad722beffdb1eb911d51bd6f72f07d304100d0743eebb"
Oct 30 22:58:14.524014 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:58:14.523991    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1c9192cb7a4940d27bad722beffdb1eb911d51bd6f72f07d304100d0743eebb\": container with ID starting with b1c9192cb7a4940d27bad722beffdb1eb911d51bd6f72f07d304100d0743eebb not found: ID does not exist" containerID="b1c9192cb7a4940d27bad722beffdb1eb911d51bd6f72f07d304100d0743eebb"
Oct 30 22:58:14.524094 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.524023    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1c9192cb7a4940d27bad722beffdb1eb911d51bd6f72f07d304100d0743eebb"} err="failed to get container status \"b1c9192cb7a4940d27bad722beffdb1eb911d51bd6f72f07d304100d0743eebb\": rpc error: code = NotFound desc = could not find container \"b1c9192cb7a4940d27bad722beffdb1eb911d51bd6f72f07d304100d0743eebb\": container with ID starting with b1c9192cb7a4940d27bad722beffdb1eb911d51bd6f72f07d304100d0743eebb not found: ID does not exist"
Oct 30 22:58:14.553407 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.553368    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh"]
Oct 30 22:58:14.565932 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.565897    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f88979-7dd56f4c95-jnwkh"]
Oct 30 22:58:14.602544 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.602504    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv"]
Oct 30 22:58:14.606169 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.606142    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-844f98df9f-d7zsv"]
Oct 30 22:58:14.776459 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.776383    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b7bae51-8941-42f0-93a1-0a20637b3b75" path="/var/lib/kubelet/pods/7b7bae51-8941-42f0-93a1-0a20637b3b75/volumes"
Oct 30 22:58:14.776830 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:14.776815    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9" path="/var/lib/kubelet/pods/e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9/volumes"
Oct 30 22:58:15.579421 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:15.579374    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:17.660457 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:17.660420    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:18.995411 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:18.995383    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:58:19.057747 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.057705    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-kube-api-access-pglfg\" (UniqueName: \"kubernetes.io/projected/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-kube-api-access-pglfg\") pod \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") "
Oct 30 22:58:19.057965 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.057827    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-data\") pod \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") "
Oct 30 22:58:19.057965 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.057866    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-workload-socket\") pod \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") "
Oct 30 22:58:19.057965 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.057931    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-token\" (UniqueName: \"kubernetes.io/projected/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-token\") pod \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") "
Oct 30 22:58:19.058156 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.057976    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istiod-ca-cert\") pod \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") "
Oct 30 22:58:19.058215 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.058180    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-podinfo\") pod \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") "
Oct 30 22:58:19.058332 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.058266    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-workload-certs\") pod \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") "
Oct 30 22:58:19.058332 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.058314    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-envoy\") pod \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") "
Oct 30 22:58:19.058458 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.058385    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-credential-socket\") pod \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\" (UID: \"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9\") "
Oct 30 22:58:19.058649 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.058616    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istiod-ca-cert" pod "efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9" (UID: "efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:58:19.058882 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.058856    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-workload-certs" pod "efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9" (UID: "efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:19.058882 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.058877    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-credential-socket" pod "efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9" (UID: "efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:19.059171 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.059143    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-data" pod "efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9" (UID: "efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:19.059299 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.059193    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:19.059299 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.059215    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:19.059299 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.059249    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:19.059485 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.059430    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-workload-socket" pod "efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9" (UID: "efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:19.062927 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.062897    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-podinfo" pod "efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9" (UID: "efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:58:19.063047 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.062923    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-token" pod "efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9" (UID: "efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:58:19.063129 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.063110    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-kube-api-access-pglfg" pod "efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9" (UID: "efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9"). InnerVolumeSpecName "kube-api-access-pglfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:58:19.063190 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.063125    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-envoy" pod "efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9" (UID: "efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:19.142730 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.142644    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"]
Oct 30 22:58:19.149974 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.149110    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"]
Oct 30 22:58:19.160151 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.160116    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pglfg\" (UniqueName: \"kubernetes.io/projected/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-kube-api-access-pglfg\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:19.160151 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.160149    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:19.160151 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.160159    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:19.160151 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.160169    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:19.160151 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.160177    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:19.160692 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.160185    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:19.308184 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.308142    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:19.552269 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.552199    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:19.567586 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.567494    4889 generic.go:356] "Generic (PLEG): container finished" podID="efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9" containerID="b1fbe6a13a71825e5dee3039efea5f663115f26bfd42233ee2845ac6bdf56ed1" exitCode=0
Oct 30 22:58:19.567586 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.567557    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"
Oct 30 22:58:19.567833 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.567573    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd" event={"ID":"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9","Type":"ContainerDied","Data":"b1fbe6a13a71825e5dee3039efea5f663115f26bfd42233ee2845ac6bdf56ed1"}
Oct 30 22:58:19.567833 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.567642    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd" event={"ID":"efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9","Type":"ContainerDied","Data":"de97f411d1fcf832865b082d40a5f5091ac66b1c3e5bbceee6db36e7b800692a"}
Oct 30 22:58:19.567833 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.567672    4889 scope.go:122] "RemoveContainer" containerID="b1fbe6a13a71825e5dee3039efea5f663115f26bfd42233ee2845ac6bdf56ed1"
Oct 30 22:58:19.583862 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.583837    4889 scope.go:122] "RemoveContainer" containerID="472db4676bd2b485e4f867a6a49d2aad2e7babee5603cd02514bcfefb1a8bee6"
Oct 30 22:58:19.596284 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.596255    4889 scope.go:122] "RemoveContainer" containerID="b1fbe6a13a71825e5dee3039efea5f663115f26bfd42233ee2845ac6bdf56ed1"
Oct 30 22:58:19.596653 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:58:19.596623    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1fbe6a13a71825e5dee3039efea5f663115f26bfd42233ee2845ac6bdf56ed1\": container with ID starting with b1fbe6a13a71825e5dee3039efea5f663115f26bfd42233ee2845ac6bdf56ed1 not found: ID does not exist" containerID="b1fbe6a13a71825e5dee3039efea5f663115f26bfd42233ee2845ac6bdf56ed1"
Oct 30 22:58:19.596715 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.596666    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1fbe6a13a71825e5dee3039efea5f663115f26bfd42233ee2845ac6bdf56ed1"} err="failed to get container status \"b1fbe6a13a71825e5dee3039efea5f663115f26bfd42233ee2845ac6bdf56ed1\": rpc error: code = NotFound desc = could not find container \"b1fbe6a13a71825e5dee3039efea5f663115f26bfd42233ee2845ac6bdf56ed1\": container with ID starting with b1fbe6a13a71825e5dee3039efea5f663115f26bfd42233ee2845ac6bdf56ed1 not found: ID does not exist"
Oct 30 22:58:19.596715 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.596689    4889 scope.go:122] "RemoveContainer" containerID="472db4676bd2b485e4f867a6a49d2aad2e7babee5603cd02514bcfefb1a8bee6"
Oct 30 22:58:19.596955 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:58:19.596938    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"472db4676bd2b485e4f867a6a49d2aad2e7babee5603cd02514bcfefb1a8bee6\": container with ID starting with 472db4676bd2b485e4f867a6a49d2aad2e7babee5603cd02514bcfefb1a8bee6 not found: ID does not exist" containerID="472db4676bd2b485e4f867a6a49d2aad2e7babee5603cd02514bcfefb1a8bee6"
Oct 30 22:58:19.597004 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.596959    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"472db4676bd2b485e4f867a6a49d2aad2e7babee5603cd02514bcfefb1a8bee6"} err="failed to get container status \"472db4676bd2b485e4f867a6a49d2aad2e7babee5603cd02514bcfefb1a8bee6\": rpc error: code = NotFound desc = could not find container \"472db4676bd2b485e4f867a6a49d2aad2e7babee5603cd02514bcfefb1a8bee6\": container with ID starting with 472db4676bd2b485e4f867a6a49d2aad2e7babee5603cd02514bcfefb1a8bee6 not found: ID does not exist"
Oct 30 22:58:19.639389 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.639345    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"]
Oct 30 22:58:19.642982 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:19.642948    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b03b33-openshift-default-79f6954695-882qd"]
Oct 30 22:58:20.777141 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:20.777102    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9" path="/var/lib/kubelet/pods/efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9/volumes"
Oct 30 22:58:21.446247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:21.445916    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:23.154961 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:23.150665    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:25.014771 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:25.014737    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rp6t6_f32ffa1e-863c-4df9-b4d1-f80463ed099c/payload-processing/0.log"
Oct 30 22:58:25.094803 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:25.094766    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:25.210077 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:25.210051    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rp6t6_f32ffa1e-863c-4df9-b4d1-f80463ed099c/payload-processing/0.log"
Oct 30 22:58:25.299799 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:25.299714    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:26.628039 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:26.628001    4889 generic.go:356] "Generic (PLEG): container finished" podID="d2279298-a1a4-469d-a188-f41478b6a598" containerID="9289fcccac2f5ebe83a35605eccd354110ae8e9ebd91bab76186f3593754995e" exitCode=0
Oct 30 22:58:26.628524 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:26.628060    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g" event={"ID":"d2279298-a1a4-469d-a188-f41478b6a598","Type":"ContainerDied","Data":"9289fcccac2f5ebe83a35605eccd354110ae8e9ebd91bab76186f3593754995e"}
Oct 30 22:58:26.628653 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:26.628624    4889 scope.go:122] "RemoveContainer" containerID="9289fcccac2f5ebe83a35605eccd354110ae8e9ebd91bab76186f3593754995e"
Oct 30 22:58:26.630652 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:26.630617    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:58:26.630788 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:26.630732    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:58:26.630860 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:26.630787    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:58:26.996575 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:26.996533    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:27.644437 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:27.644399    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g" event={"ID":"d2279298-a1a4-469d-a188-f41478b6a598","Type":"ContainerStarted","Data":"2dd5d7e1811bbf9f5aeff1b49ba3e29f8dd3c016922fa6acf6b1f86fe5cc68af"}
Oct 30 22:58:27.790358 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:27.790329    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:58:27.791964 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:27.791933    4889 patch_prober.go:29] interesting pod/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Oct 30 22:58:27.792145 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:27.791994    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g" podUID="d2279298-a1a4-469d-a188-f41478b6a598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Oct 30 22:58:28.790659 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:28.790613    4889 patch_prober.go:29] interesting pod/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Oct 30 22:58:28.791097 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:28.790679    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g" podUID="d2279298-a1a4-469d-a188-f41478b6a598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Oct 30 22:58:28.989167 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:28.989132    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:29.790045 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:29.790001    4889 patch_prober.go:29] interesting pod/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Oct 30 22:58:29.790259 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:29.790078    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g" podUID="d2279298-a1a4-469d-a188-f41478b6a598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Oct 30 22:58:30.342194 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.342165    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:30.535266 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.535191    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:30.677277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.677158    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n"]
Oct 30 22:58:30.678292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.678262    4889 cpu_manager.go:414] "containerMap: removing container" podUID="efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9" containerName="istio-proxy"
Oct 30 22:58:30.678292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.678292    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9" containerName="istio-proxy"
Oct 30 22:58:30.678517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.678327    4889 cpu_manager.go:414] "containerMap: removing container" podUID="2f8596b1-c420-46cd-85cd-6b68dffff980" containerName="main"
Oct 30 22:58:30.678517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.678335    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f8596b1-c420-46cd-85cd-6b68dffff980" containerName="main"
Oct 30 22:58:30.678517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.678382    4889 cpu_manager.go:414] "containerMap: removing container" podUID="efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9" containerName="istio-proxy"
Oct 30 22:58:30.678517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.678390    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9" containerName="istio-proxy"
Oct 30 22:58:30.678517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.678407    4889 cpu_manager.go:414] "containerMap: removing container" podUID="7b7bae51-8941-42f0-93a1-0a20637b3b75" containerName="payload-processing"
Oct 30 22:58:30.678517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.678414    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b7bae51-8941-42f0-93a1-0a20637b3b75" containerName="payload-processing"
Oct 30 22:58:30.678517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.678433    4889 cpu_manager.go:414] "containerMap: removing container" podUID="5a707459-e53d-486b-a1e5-0456ab08570c" containerName="istio-proxy"
Oct 30 22:58:30.678517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.678440    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a707459-e53d-486b-a1e5-0456ab08570c" containerName="istio-proxy"
Oct 30 22:58:30.678517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.678451    4889 cpu_manager.go:414] "containerMap: removing container" podUID="512c5c51-57a4-48d3-b1c3-e1fdb7bbd483" containerName="payload-processing"
Oct 30 22:58:30.678517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.678458    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="512c5c51-57a4-48d3-b1c3-e1fdb7bbd483" containerName="payload-processing"
Oct 30 22:58:30.678517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.678483    4889 cpu_manager.go:414] "containerMap: removing container" podUID="0997827b-80e4-4077-a22f-3d2026492158" containerName="maas-api"
Oct 30 22:58:30.678517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.678491    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0997827b-80e4-4077-a22f-3d2026492158" containerName="maas-api"
Oct 30 22:58:30.678517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.678510    4889 cpu_manager.go:414] "containerMap: removing container" podUID="e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9" containerName="payload-pre-processing"
Oct 30 22:58:30.678517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.678518    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9" containerName="payload-pre-processing"
Oct 30 22:58:30.679376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.678706    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9" containerName="istio-proxy"
Oct 30 22:58:30.679376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.678721    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b7bae51-8941-42f0-93a1-0a20637b3b75" containerName="payload-processing"
Oct 30 22:58:30.679376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.678735    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="512c5c51-57a4-48d3-b1c3-e1fdb7bbd483" containerName="payload-processing"
Oct 30 22:58:30.679376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.678750    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="0997827b-80e4-4077-a22f-3d2026492158" containerName="maas-api"
Oct 30 22:58:30.679376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.678760    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a707459-e53d-486b-a1e5-0456ab08570c" containerName="istio-proxy"
Oct 30 22:58:30.679376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.678767    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0a9064e-4af4-4e01-8c3f-d7e58d78c5a9" containerName="payload-pre-processing"
Oct 30 22:58:30.679376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.678781    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f8596b1-c420-46cd-85cd-6b68dffff980" containerName="main"
Oct 30 22:58:30.701553 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.699488    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n"]
Oct 30 22:58:30.701900 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.701877    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n"
Oct 30 22:58:30.737311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.725457    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f0a807\""
Oct 30 22:58:30.755283 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.742541    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f0a807-dockercfg-64l5v\""
Oct 30 22:58:30.786616 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.786330    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw84q\" (UniqueName: \"kubernetes.io/projected/3d913d76-fad1-4092-bab7-e13c74cd2e0f-kube-api-access-tw84q\") pod \"payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n\" (UID: \"3d913d76-fad1-4092-bab7-e13c74cd2e0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n"
Oct 30 22:58:30.786822 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.786691    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d913d76-fad1-4092-bab7-e13c74cd2e0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n\" (UID: \"3d913d76-fad1-4092-bab7-e13c74cd2e0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n"
Oct 30 22:58:30.790161 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.790120    4889 patch_prober.go:29] interesting pod/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Oct 30 22:58:30.790799 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.790756    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g" podUID="d2279298-a1a4-469d-a188-f41478b6a598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Oct 30 22:58:30.840922 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.840867    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m"]
Oct 30 22:58:30.841937 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.841916    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="efb5b6d8-8552-46dc-9ef3-26f8fd64a8b9" containerName="istio-proxy"
Oct 30 22:58:30.845880 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.845855    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m"
Oct 30 22:58:30.861332 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.861300    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m"]
Oct 30 22:58:30.900369 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.900321    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw84q\" (UniqueName: \"kubernetes.io/projected/3d913d76-fad1-4092-bab7-e13c74cd2e0f-kube-api-access-tw84q\") pod \"payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n\" (UID: \"3d913d76-fad1-4092-bab7-e13c74cd2e0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n"
Oct 30 22:58:30.900603 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.900401    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z6sf\" (UniqueName: \"kubernetes.io/projected/8cf81220-6bf3-42ab-8330-ed9a73dc8190-kube-api-access-5z6sf\") pod \"payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m\" (UID: \"8cf81220-6bf3-42ab-8330-ed9a73dc8190\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m"
Oct 30 22:58:30.900603 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.900459    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d913d76-fad1-4092-bab7-e13c74cd2e0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n\" (UID: \"3d913d76-fad1-4092-bab7-e13c74cd2e0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n"
Oct 30 22:58:30.900779 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.900735    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cf81220-6bf3-42ab-8330-ed9a73dc8190-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m\" (UID: \"8cf81220-6bf3-42ab-8330-ed9a73dc8190\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m"
Oct 30 22:58:30.901494 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.901463    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d913d76-fad1-4092-bab7-e13c74cd2e0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n\" (UID: \"3d913d76-fad1-4092-bab7-e13c74cd2e0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n"
Oct 30 22:58:30.938320 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:30.938209    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw84q\" (UniqueName: \"kubernetes.io/projected/3d913d76-fad1-4092-bab7-e13c74cd2e0f-kube-api-access-tw84q\") pod \"payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n\" (UID: \"3d913d76-fad1-4092-bab7-e13c74cd2e0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n"
Oct 30 22:58:31.002320 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:31.002281    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cf81220-6bf3-42ab-8330-ed9a73dc8190-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m\" (UID: \"8cf81220-6bf3-42ab-8330-ed9a73dc8190\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m"
Oct 30 22:58:31.002559 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:31.002392    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z6sf\" (UniqueName: \"kubernetes.io/projected/8cf81220-6bf3-42ab-8330-ed9a73dc8190-kube-api-access-5z6sf\") pod \"payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m\" (UID: \"8cf81220-6bf3-42ab-8330-ed9a73dc8190\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m"
Oct 30 22:58:31.003067 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:31.003046    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cf81220-6bf3-42ab-8330-ed9a73dc8190-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m\" (UID: \"8cf81220-6bf3-42ab-8330-ed9a73dc8190\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m"
Oct 30 22:58:31.036944 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:31.036841    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z6sf\" (UniqueName: \"kubernetes.io/projected/8cf81220-6bf3-42ab-8330-ed9a73dc8190-kube-api-access-5z6sf\") pod \"payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m\" (UID: \"8cf81220-6bf3-42ab-8330-ed9a73dc8190\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m"
Oct 30 22:58:31.089187 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:31.089139    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n"
Oct 30 22:58:31.162858 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:31.162820    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m"
Oct 30 22:58:31.472778 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:31.472739    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:58:31.488053 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:58:31.488008    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d913d76_fad1_4092_bab7_e13c74cd2e0f.slice/crio-2f5975e2a02fe503244e3700f0dae1894d1f2602a1461cbcc2b10216e604ae9d WatchSource:0}: Error finding container 2f5975e2a02fe503244e3700f0dae1894d1f2602a1461cbcc2b10216e604ae9d: Status 404 returned error can't find the container with id 2f5975e2a02fe503244e3700f0dae1894d1f2602a1461cbcc2b10216e604ae9d
Oct 30 22:58:31.488406 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:31.488384    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n"]
Oct 30 22:58:31.580639 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:58:31.580596    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf81220_6bf3_42ab_8330_ed9a73dc8190.slice/crio-ad4a0e4e8abd738c9deb7632c43efb39b1190415c4b084493bc6bf7ea49db92b WatchSource:0}: Error finding container ad4a0e4e8abd738c9deb7632c43efb39b1190415c4b084493bc6bf7ea49db92b: Status 404 returned error can't find the container with id ad4a0e4e8abd738c9deb7632c43efb39b1190415c4b084493bc6bf7ea49db92b
Oct 30 22:58:31.580930 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:31.580898    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m"]
Oct 30 22:58:31.650125 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:31.650088    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:31.692437 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:31.692388    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n" event={"ID":"3d913d76-fad1-4092-bab7-e13c74cd2e0f","Type":"ContainerStarted","Data":"2f5975e2a02fe503244e3700f0dae1894d1f2602a1461cbcc2b10216e604ae9d"}
Oct 30 22:58:31.693861 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:31.693835    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m" event={"ID":"8cf81220-6bf3-42ab-8330-ed9a73dc8190","Type":"ContainerStarted","Data":"ad4a0e4e8abd738c9deb7632c43efb39b1190415c4b084493bc6bf7ea49db92b"}
Oct 30 22:58:31.790008 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:31.789965    4889 patch_prober.go:29] interesting pod/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Oct 30 22:58:31.790270 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:31.790024    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g" podUID="d2279298-a1a4-469d-a188-f41478b6a598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Oct 30 22:58:32.763450 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:32.763415    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"]
Oct 30 22:58:32.769160 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:32.769073    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"
Oct 30 22:58:32.788266 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:32.788210    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"]
Oct 30 22:58:32.792116 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:32.792071    4889 patch_prober.go:29] interesting pod/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Oct 30 22:58:32.792331 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:32.792143    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g" podUID="d2279298-a1a4-469d-a188-f41478b6a598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Oct 30 22:58:32.797271 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:32.797084    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-bed544\"/\"kube-root-ca.crt\""
Oct 30 22:58:32.797271 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:32.797149    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-bed544\"/\"openshift-service-ca.crt\""
Oct 30 22:58:32.797271 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:32.797248    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-bed544\"/\"default-dockercfg-xg4wc\""
Oct 30 22:58:32.802591 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:32.802545    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-bed544\"/\"sub-test-model-bed544-kserve-self-signed-certs\""
Oct 30 22:58:32.882331 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:32.882214    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-tmp-dir\") pod \"sub-test-model-bed544-kserve-746c6658fc-qtgw6\" (UID: \"1cd0f933-b208-424d-a9c1-04f8c6044054\") " pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"
Oct 30 22:58:32.882675 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:32.882651    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-home\") pod \"sub-test-model-bed544-kserve-746c6658fc-qtgw6\" (UID: \"1cd0f933-b208-424d-a9c1-04f8c6044054\") " pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"
Oct 30 22:58:32.882913 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:32.882899    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-dshm\") pod \"sub-test-model-bed544-kserve-746c6658fc-qtgw6\" (UID: \"1cd0f933-b208-424d-a9c1-04f8c6044054\") " pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"
Oct 30 22:58:32.883057 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:32.883046    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cd0f933-b208-424d-a9c1-04f8c6044054-tls-certs\") pod \"sub-test-model-bed544-kserve-746c6658fc-qtgw6\" (UID: \"1cd0f933-b208-424d-a9c1-04f8c6044054\") " pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"
Oct 30 22:58:32.883677 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:32.883495    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hr7k\" (UniqueName: \"kubernetes.io/projected/1cd0f933-b208-424d-a9c1-04f8c6044054-kube-api-access-8hr7k\") pod \"sub-test-model-bed544-kserve-746c6658fc-qtgw6\" (UID: \"1cd0f933-b208-424d-a9c1-04f8c6044054\") " pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"
Oct 30 22:58:32.883863 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:32.883843    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-model-cache\") pod \"sub-test-model-bed544-kserve-746c6658fc-qtgw6\" (UID: \"1cd0f933-b208-424d-a9c1-04f8c6044054\") " pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"
Oct 30 22:58:32.945180 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:32.945116    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:32.994316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:32.985880    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-tmp-dir\") pod \"sub-test-model-bed544-kserve-746c6658fc-qtgw6\" (UID: \"1cd0f933-b208-424d-a9c1-04f8c6044054\") " pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"
Oct 30 22:58:32.994316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:32.985931    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-home\") pod \"sub-test-model-bed544-kserve-746c6658fc-qtgw6\" (UID: \"1cd0f933-b208-424d-a9c1-04f8c6044054\") " pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"
Oct 30 22:58:32.994316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:32.985967    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-dshm\") pod \"sub-test-model-bed544-kserve-746c6658fc-qtgw6\" (UID: \"1cd0f933-b208-424d-a9c1-04f8c6044054\") " pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"
Oct 30 22:58:32.994316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:32.986016    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cd0f933-b208-424d-a9c1-04f8c6044054-tls-certs\") pod \"sub-test-model-bed544-kserve-746c6658fc-qtgw6\" (UID: \"1cd0f933-b208-424d-a9c1-04f8c6044054\") " pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"
Oct 30 22:58:32.994316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:32.986099    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hr7k\" (UniqueName: \"kubernetes.io/projected/1cd0f933-b208-424d-a9c1-04f8c6044054-kube-api-access-8hr7k\") pod \"sub-test-model-bed544-kserve-746c6658fc-qtgw6\" (UID: \"1cd0f933-b208-424d-a9c1-04f8c6044054\") " pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"
Oct 30 22:58:32.994316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:32.986154    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-model-cache\") pod \"sub-test-model-bed544-kserve-746c6658fc-qtgw6\" (UID: \"1cd0f933-b208-424d-a9c1-04f8c6044054\") " pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"
Oct 30 22:58:32.994316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:32.986670    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-model-cache\") pod \"sub-test-model-bed544-kserve-746c6658fc-qtgw6\" (UID: \"1cd0f933-b208-424d-a9c1-04f8c6044054\") " pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"
Oct 30 22:58:32.994316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:32.986922    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-tmp-dir\") pod \"sub-test-model-bed544-kserve-746c6658fc-qtgw6\" (UID: \"1cd0f933-b208-424d-a9c1-04f8c6044054\") " pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"
Oct 30 22:58:32.994316 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:32.987161    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-home\") pod \"sub-test-model-bed544-kserve-746c6658fc-qtgw6\" (UID: \"1cd0f933-b208-424d-a9c1-04f8c6044054\") " pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"
Oct 30 22:58:33.001934 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:33.001897    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-dshm\") pod \"sub-test-model-bed544-kserve-746c6658fc-qtgw6\" (UID: \"1cd0f933-b208-424d-a9c1-04f8c6044054\") " pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"
Oct 30 22:58:33.006893 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:33.006825    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cd0f933-b208-424d-a9c1-04f8c6044054-tls-certs\") pod \"sub-test-model-bed544-kserve-746c6658fc-qtgw6\" (UID: \"1cd0f933-b208-424d-a9c1-04f8c6044054\") " pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"
Oct 30 22:58:33.029720 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:33.029572    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hr7k\" (UniqueName: \"kubernetes.io/projected/1cd0f933-b208-424d-a9c1-04f8c6044054-kube-api-access-8hr7k\") pod \"sub-test-model-bed544-kserve-746c6658fc-qtgw6\" (UID: \"1cd0f933-b208-424d-a9c1-04f8c6044054\") " pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"
Oct 30 22:58:33.103634 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:33.103338    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"
Oct 30 22:58:33.470459 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:58:33.470413    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cd0f933_b208_424d_a9c1_04f8c6044054.slice/crio-30fce40e1fe792553a8db1d6c3857df1924886d34b5ac68df0bb4e5abf55dc6b WatchSource:0}: Error finding container 30fce40e1fe792553a8db1d6c3857df1924886d34b5ac68df0bb4e5abf55dc6b: Status 404 returned error can't find the container with id 30fce40e1fe792553a8db1d6c3857df1924886d34b5ac68df0bb4e5abf55dc6b
Oct 30 22:58:33.471976 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:33.471943    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"]
Oct 30 22:58:33.560888 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:33.560845    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:33.760521 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:33.760476    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6" event={"ID":"1cd0f933-b208-424d-a9c1-04f8c6044054","Type":"ContainerStarted","Data":"df760719ade8961ccddef5d9380fd5a3e91c64a7215562318e68bba5457d7e05"}
Oct 30 22:58:33.760737 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:33.760534    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6" event={"ID":"1cd0f933-b208-424d-a9c1-04f8c6044054","Type":"ContainerStarted","Data":"30fce40e1fe792553a8db1d6c3857df1924886d34b5ac68df0bb4e5abf55dc6b"}
Oct 30 22:58:33.797582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:33.797546    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:58:33.799122 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:33.799092    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 22:58:33.808695 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:33.808609    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6" podStartSLOduration=1.808591871 podStartE2EDuration="1.808591871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:58:33.803373357 +0000 UTC m=+3099.159739673" watchObservedRunningTime="2026-10-30 22:58:33.808591871 +0000 UTC m=+3099.164958194"
Oct 30 22:58:34.103738 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.103656    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"
Oct 30 22:58:34.173951 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.173894    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main" containerID="cri-o://652b969d08f26d003c44a6d38d3b06b46778fd8d2cb3c79249a8c75f00cc43ab" gracePeriod=45
Oct 30 22:58:34.174671 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.174625    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main" containerID="cri-o://9979ecfa6ebabfae18e7be2303f8813d1b85e720a9fc2de0b763b034069a322f" gracePeriod=45
Oct 30 22:58:34.497268 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.496837    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:58:34.527211 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.527172    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:58:34.624810 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.624713    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42e72bf1-014e-446f-902a-e3a8ba06d34b-kube-api-access-5f4xs\" (UniqueName: \"kubernetes.io/projected/42e72bf1-014e-446f-902a-e3a8ba06d34b-kube-api-access-5f4xs\") pod \"42e72bf1-014e-446f-902a-e3a8ba06d34b\" (UID: \"42e72bf1-014e-446f-902a-e3a8ba06d34b\") "
Oct 30 22:58:34.624810 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.624775    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-tmp-dir\") pod \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\" (UID: \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\") "
Oct 30 22:58:34.625059 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.624911    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42e72bf1-014e-446f-902a-e3a8ba06d34b-tls-certs\" (UniqueName: \"kubernetes.io/secret/42e72bf1-014e-446f-902a-e3a8ba06d34b-tls-certs\") pod \"42e72bf1-014e-446f-902a-e3a8ba06d34b\" (UID: \"42e72bf1-014e-446f-902a-e3a8ba06d34b\") "
Oct 30 22:58:34.625059 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.624985    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-tls-certs\") pod \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\" (UID: \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\") "
Oct 30 22:58:34.625188 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.625151    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-model-cache\") pod \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\" (UID: \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\") "
Oct 30 22:58:34.625287 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.625268    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-tmp-dir\") pod \"42e72bf1-014e-446f-902a-e3a8ba06d34b\" (UID: \"42e72bf1-014e-446f-902a-e3a8ba06d34b\") "
Oct 30 22:58:34.625349 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.625323    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-home\" (UniqueName: \"kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-home\") pod \"42e72bf1-014e-446f-902a-e3a8ba06d34b\" (UID: \"42e72bf1-014e-446f-902a-e3a8ba06d34b\") "
Oct 30 22:58:34.625415 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.625374    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-model-cache\") pod \"42e72bf1-014e-446f-902a-e3a8ba06d34b\" (UID: \"42e72bf1-014e-446f-902a-e3a8ba06d34b\") "
Oct 30 22:58:34.625415 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.625404    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-dshm\") pod \"42e72bf1-014e-446f-902a-e3a8ba06d34b\" (UID: \"42e72bf1-014e-446f-902a-e3a8ba06d34b\") "
Oct 30 22:58:34.625536 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.625463    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-home\" (UniqueName: \"kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-home\") pod \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\" (UID: \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\") "
Oct 30 22:58:34.625595 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.625569    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-dshm\") pod \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\" (UID: \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\") "
Oct 30 22:58:34.625731 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.625712    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-kube-api-access-pzlmh\" (UniqueName: \"kubernetes.io/projected/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-kube-api-access-pzlmh\") pod \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\" (UID: \"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05\") "
Oct 30 22:58:34.627403 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.626136    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-model-cache" pod "42e72bf1-014e-446f-902a-e3a8ba06d34b" (UID: "42e72bf1-014e-446f-902a-e3a8ba06d34b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:34.627403 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.626432    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-home" pod "42e72bf1-014e-446f-902a-e3a8ba06d34b" (UID: "42e72bf1-014e-446f-902a-e3a8ba06d34b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:34.627403 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.626478    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-tmp-dir" pod "42e72bf1-014e-446f-902a-e3a8ba06d34b" (UID: "42e72bf1-014e-446f-902a-e3a8ba06d34b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:34.627403 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.626535    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:34.627403 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.626567    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-model-cache" pod "5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" (UID: "5f4c1b10-0d9c-4bb7-a49a-df55737c6a05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:34.627403 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.626786    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-home" pod "5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" (UID: "5f4c1b10-0d9c-4bb7-a49a-df55737c6a05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:34.627403 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.627343    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-tmp-dir" pod "5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" (UID: "5f4c1b10-0d9c-4bb7-a49a-df55737c6a05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:34.649750 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.634493    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42e72bf1-014e-446f-902a-e3a8ba06d34b-tls-certs" pod "42e72bf1-014e-446f-902a-e3a8ba06d34b" (UID: "42e72bf1-014e-446f-902a-e3a8ba06d34b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:58:34.649750 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.634604    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42e72bf1-014e-446f-902a-e3a8ba06d34b-kube-api-access-5f4xs" pod "42e72bf1-014e-446f-902a-e3a8ba06d34b" (UID: "42e72bf1-014e-446f-902a-e3a8ba06d34b"). InnerVolumeSpecName "kube-api-access-5f4xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:58:34.649750 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.634789    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-dshm" pod "42e72bf1-014e-446f-902a-e3a8ba06d34b" (UID: "42e72bf1-014e-446f-902a-e3a8ba06d34b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:34.649750 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.635391    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-dshm" pod "5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" (UID: "5f4c1b10-0d9c-4bb7-a49a-df55737c6a05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:34.649750 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.635840    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-kube-api-access-pzlmh" pod "5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" (UID: "5f4c1b10-0d9c-4bb7-a49a-df55737c6a05"). InnerVolumeSpecName "kube-api-access-pzlmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:58:34.649750 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.636753    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-tls-certs" pod "5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" (UID: "5f4c1b10-0d9c-4bb7-a49a-df55737c6a05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:58:34.728434 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.728397    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:34.728434 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.728428    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:34.728434 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.728437    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:34.728434 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.728445    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:34.728822 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.728453    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42e72bf1-014e-446f-902a-e3a8ba06d34b-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:34.728822 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.728461    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:34.728822 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.728468    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:34.728822 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.728476    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzlmh\" (UniqueName: \"kubernetes.io/projected/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-kube-api-access-pzlmh\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:34.728822 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.728485    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f4xs\" (UniqueName: \"kubernetes.io/projected/42e72bf1-014e-446f-902a-e3a8ba06d34b-kube-api-access-5f4xs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:34.728822 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.728493    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:34.728822 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.728504    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42e72bf1-014e-446f-902a-e3a8ba06d34b-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:34.774545 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.774500    4889 generic.go:356] "Generic (PLEG): container finished" podID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerID="652b969d08f26d003c44a6d38d3b06b46778fd8d2cb3c79249a8c75f00cc43ab" exitCode=0
Oct 30 22:58:34.774738 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.774603    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"
Oct 30 22:58:34.782309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.779001    4889 generic.go:356] "Generic (PLEG): container finished" podID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerID="9979ecfa6ebabfae18e7be2303f8813d1b85e720a9fc2de0b763b034069a322f" exitCode=0
Oct 30 22:58:34.782309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.779101    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:58:34.782309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.780573    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n"
Oct 30 22:58:34.782309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.780623    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" event={"ID":"42e72bf1-014e-446f-902a-e3a8ba06d34b","Type":"ContainerDied","Data":"652b969d08f26d003c44a6d38d3b06b46778fd8d2cb3c79249a8c75f00cc43ab"}
Oct 30 22:58:34.782309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.780658    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f" event={"ID":"42e72bf1-014e-446f-902a-e3a8ba06d34b","Type":"ContainerDied","Data":"2d816b85ebe81c7aaf82eb8c1d97a771aa557bb2cbc6fdb8319fa5ffe0c1f7fd"}
Oct 30 22:58:34.782309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.780673    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n" event={"ID":"3d913d76-fad1-4092-bab7-e13c74cd2e0f","Type":"ContainerStarted","Data":"c31e2367c7a7cf2738859b3d7ab70c05d6dc49704f5b99bdc255d84d9efb78ca"}
Oct 30 22:58:34.782309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.780696    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" event={"ID":"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05","Type":"ContainerDied","Data":"9979ecfa6ebabfae18e7be2303f8813d1b85e720a9fc2de0b763b034069a322f"}
Oct 30 22:58:34.782309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.780714    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" event={"ID":"5f4c1b10-0d9c-4bb7-a49a-df55737c6a05","Type":"ContainerDied","Data":"6671f0e60d3606eab5f5a826041f23bf5261a01ec88a74c0e4ea5e7296e19a7f"}
Oct 30 22:58:34.782309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.780736    4889 scope.go:122] "RemoveContainer" containerID="652b969d08f26d003c44a6d38d3b06b46778fd8d2cb3c79249a8c75f00cc43ab"
Oct 30 22:58:34.788580 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.788545    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m" event={"ID":"8cf81220-6bf3-42ab-8330-ed9a73dc8190","Type":"ContainerStarted","Data":"684e20da2d668adb6f55655eb527fdac9a36412932867ef09a81cfdf3d6baf43"}
Oct 30 22:58:34.789728 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.789701    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m"
Oct 30 22:58:34.800028 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.800003    4889 scope.go:122] "RemoveContainer" containerID="652b969d08f26d003c44a6d38d3b06b46778fd8d2cb3c79249a8c75f00cc43ab"
Oct 30 22:58:34.800484 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:58:34.800458    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"652b969d08f26d003c44a6d38d3b06b46778fd8d2cb3c79249a8c75f00cc43ab\": container with ID starting with 652b969d08f26d003c44a6d38d3b06b46778fd8d2cb3c79249a8c75f00cc43ab not found: ID does not exist" containerID="652b969d08f26d003c44a6d38d3b06b46778fd8d2cb3c79249a8c75f00cc43ab"
Oct 30 22:58:34.800537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.800499    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"652b969d08f26d003c44a6d38d3b06b46778fd8d2cb3c79249a8c75f00cc43ab"} err="failed to get container status \"652b969d08f26d003c44a6d38d3b06b46778fd8d2cb3c79249a8c75f00cc43ab\": rpc error: code = NotFound desc = could not find container \"652b969d08f26d003c44a6d38d3b06b46778fd8d2cb3c79249a8c75f00cc43ab\": container with ID starting with 652b969d08f26d003c44a6d38d3b06b46778fd8d2cb3c79249a8c75f00cc43ab not found: ID does not exist"
Oct 30 22:58:34.800537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.800527    4889 scope.go:122] "RemoveContainer" containerID="9979ecfa6ebabfae18e7be2303f8813d1b85e720a9fc2de0b763b034069a322f"
Oct 30 22:58:34.817935 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.817908    4889 scope.go:122] "RemoveContainer" containerID="9979ecfa6ebabfae18e7be2303f8813d1b85e720a9fc2de0b763b034069a322f"
Oct 30 22:58:34.818389 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:58:34.818359    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9979ecfa6ebabfae18e7be2303f8813d1b85e720a9fc2de0b763b034069a322f\": container with ID starting with 9979ecfa6ebabfae18e7be2303f8813d1b85e720a9fc2de0b763b034069a322f not found: ID does not exist" containerID="9979ecfa6ebabfae18e7be2303f8813d1b85e720a9fc2de0b763b034069a322f"
Oct 30 22:58:34.818517 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.818402    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9979ecfa6ebabfae18e7be2303f8813d1b85e720a9fc2de0b763b034069a322f"} err="failed to get container status \"9979ecfa6ebabfae18e7be2303f8813d1b85e720a9fc2de0b763b034069a322f\": rpc error: code = NotFound desc = could not find container \"9979ecfa6ebabfae18e7be2303f8813d1b85e720a9fc2de0b763b034069a322f\": container with ID starting with 9979ecfa6ebabfae18e7be2303f8813d1b85e720a9fc2de0b763b034069a322f not found: ID does not exist"
Oct 30 22:58:34.840085 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.840036    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"]
Oct 30 22:58:34.844101 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.844072    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0ad7c5/distinct-w4-0ad7c5-kserve-796f56b5ff-zl65f"]
Oct 30 22:58:34.882850 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.882725    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m" podStartSLOduration=2.309966448 podStartE2EDuration="4.882709369s" totalImagesPullingTime="2.572742921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:58:34.881573749 +0000 UTC m=+3100.237940067" watchObservedRunningTime="2026-10-30 22:58:34.882709369 +0000 UTC m=+3100.239075683"
Oct 30 22:58:34.930141 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:34.930029    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n" podStartSLOduration=2.270962717 podStartE2EDuration="4.930012172s" totalImagesPullingTime="2.659049455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:58:34.926858494 +0000 UTC m=+3100.283224812" watchObservedRunningTime="2026-10-30 22:58:34.930012172 +0000 UTC m=+3100.286378522"
Oct 30 22:58:35.375196 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.375152    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"]
Oct 30 22:58:35.376268 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.376247    4889 cpu_manager.go:414] "containerMap: removing container" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main"
Oct 30 22:58:35.376337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.376269    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main"
Oct 30 22:58:35.376337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.376317    4889 cpu_manager.go:414] "containerMap: removing container" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main"
Oct 30 22:58:35.376337 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.376325    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main"
Oct 30 22:58:35.376537 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.376525    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" containerName="main"
Oct 30 22:58:35.376576 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.376545    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" containerName="main"
Oct 30 22:58:35.382420 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.382391    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.384868 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.384840    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:35.396920 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.396891    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"]
Oct 30 22:58:35.400147 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.400116    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-510f80-openshift-default-dockercfg-w2m4c\""
Oct 30 22:58:35.438147 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.438082    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-token\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.438404 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.438175    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1cef895-22cd-4599-8239-d2ebf69cd234-istiod-ca-cert\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.438404 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.438266    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-data\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.438404 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.438318    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-workload-certs\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.438404 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.438354    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-podinfo\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.438690 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.438449    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zdtt\" (UniqueName: \"kubernetes.io/projected/d1cef895-22cd-4599-8239-d2ebf69cd234-kube-api-access-8zdtt\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.438690 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.438512    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-workload-socket\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.438690 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.438542    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-envoy\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.438690 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.438566    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-credential-socket\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.539641 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.539604    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1cef895-22cd-4599-8239-d2ebf69cd234-istiod-ca-cert\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.539842 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.539663    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-data\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.539842 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.539708    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-workload-certs\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.539842 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.539743    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-podinfo\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.539842 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.539835    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zdtt\" (UniqueName: \"kubernetes.io/projected/d1cef895-22cd-4599-8239-d2ebf69cd234-kube-api-access-8zdtt\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.540173 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.539901    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-workload-socket\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.540173 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.539930    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-envoy\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.540173 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.539952    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-credential-socket\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.540173 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.540085    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-token\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.540419 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.540321    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-workload-certs\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.540585 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.540560    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1cef895-22cd-4599-8239-d2ebf69cd234-istiod-ca-cert\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.540662 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.540571    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-credential-socket\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.540801 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.540781    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-data\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.540864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.540845    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-workload-socket\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.543180 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.543148    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-envoy\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.543818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.543793    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-podinfo\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.562054 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.562010    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-token\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.562238 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.562138    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zdtt\" (UniqueName: \"kubernetes.io/projected/d1cef895-22cd-4599-8239-d2ebf69cd234-kube-api-access-8zdtt\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:35.706039 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:35.705949    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:36.035946 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:58:36.035905    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1cef895_22cd_4599_8239_d2ebf69cd234.slice/crio-a20252d458c309611ec5f90a2be55fe98b1b49484bcf21030846f64729a3e933 WatchSource:0}: Error finding container a20252d458c309611ec5f90a2be55fe98b1b49484bcf21030846f64729a3e933: Status 404 returned error can't find the container with id a20252d458c309611ec5f90a2be55fe98b1b49484bcf21030846f64729a3e933
Oct 30 22:58:36.036442 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:36.036067    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"]
Oct 30 22:58:36.039380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:36.039341    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:58:36.039503 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:36.039417    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:58:36.039503 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:36.039446    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:58:36.488404 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:36.481361    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"]
Oct 30 22:58:36.574592 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:36.574555    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:36.776336 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:36.776302    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42e72bf1-014e-446f-902a-e3a8ba06d34b" path="/var/lib/kubelet/pods/42e72bf1-014e-446f-902a-e3a8ba06d34b/volumes"
Oct 30 22:58:36.814052 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:36.814000    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf" event={"ID":"d1cef895-22cd-4599-8239-d2ebf69cd234","Type":"ContainerStarted","Data":"7fa6f3f6db1f4c704a6ae567eff11c13eb96b27adc7cfb26a8216f5aa93cde38"}
Oct 30 22:58:36.814252 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:36.814063    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf" event={"ID":"d1cef895-22cd-4599-8239-d2ebf69cd234","Type":"ContainerStarted","Data":"a20252d458c309611ec5f90a2be55fe98b1b49484bcf21030846f64729a3e933"}
Oct 30 22:58:36.864677 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:36.864610    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf" podStartSLOduration=1.8645940269999999 podStartE2EDuration="1.864594027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:58:36.861931455 +0000 UTC m=+3102.218297769" watchObservedRunningTime="2026-10-30 22:58:36.864594027 +0000 UTC m=+3102.220960344"
Oct 30 22:58:37.707528 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:37.706545    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:37.709129 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:37.709091    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 22:58:37.728330 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:37.709179    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf" podUID="d1cef895-22cd-4599-8239-d2ebf69cd234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 22:58:37.754717 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:37.754675    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:38.706388 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:38.706350    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 22:58:38.706590 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:38.706447    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf" podUID="d1cef895-22cd-4599-8239-d2ebf69cd234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 22:58:39.580116 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:39.580079    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:39.706443 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:39.706406    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 22:58:39.706632 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:39.706462    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf" podUID="d1cef895-22cd-4599-8239-d2ebf69cd234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 22:58:40.304871 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:58:40.304823    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:58:40.706911 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:40.706813    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 22:58:40.706911 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:40.706879    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf" podUID="d1cef895-22cd-4599-8239-d2ebf69cd234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 22:58:41.383458 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:41.383410    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:41.706917 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:41.706827    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 22:58:41.706917 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:41.706898    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf" podUID="d1cef895-22cd-4599-8239-d2ebf69cd234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 22:58:42.707108 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:42.707066    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 22:58:42.707688 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:42.707171    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf" podUID="d1cef895-22cd-4599-8239-d2ebf69cd234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 22:58:43.104804 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:43.104752    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"
Oct 30 22:58:43.124716 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:43.124666    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"
Oct 30 22:58:43.222705 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:43.222667    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:43.712888 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:43.712853    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:43.713378 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:43.713359    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:43.714071 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:43.714053    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 22:58:43.906277 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:43.905480    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"
Oct 30 22:58:45.170415 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:45.170378    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:45.358416 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:45.358375    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:45.806426 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:45.806392    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n"
Oct 30 22:58:45.806605 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:45.806477    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m"
Oct 30 22:58:47.865052 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:47.865010    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:48.311560 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.311506    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:48.766167 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.766125    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"]
Oct 30 22:58:48.774572 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.774535    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:48.799548 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.799516    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"]
Oct 30 22:58:48.807861 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.807827    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-76d54a-openshift-default-dockercfg-6s6p4\""
Oct 30 22:58:48.874735 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.874697    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:48.883551 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.882698    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-podinfo\") pod \"e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") " pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:48.883551 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.882833    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istiod-ca-cert\") pod \"e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") " pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:48.883551 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.882888    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-token\") pod \"e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") " pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:48.883551 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.883032    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdnbh\" (UniqueName: \"kubernetes.io/projected/1aa5f50b-7768-46fd-9ce4-76c47268bce2-kube-api-access-vdnbh\") pod \"e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") " pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:48.883551 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.883105    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-data\") pod \"e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") " pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:48.883551 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.883148    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-envoy\") pod \"e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") " pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:48.883551 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.883210    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-workload-socket\") pod \"e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") " pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:48.883551 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.883317    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-workload-certs\") pod \"e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") " pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:48.883551 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.883410    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-credential-socket\") pod \"e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") " pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:48.988248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.987998    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-podinfo\") pod \"e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") " pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:48.988248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.988105    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istiod-ca-cert\") pod \"e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") " pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:48.988248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.988149    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-token\") pod \"e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") " pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:48.988248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.988254    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdnbh\" (UniqueName: \"kubernetes.io/projected/1aa5f50b-7768-46fd-9ce4-76c47268bce2-kube-api-access-vdnbh\") pod \"e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") " pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:48.988678 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.988314    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-data\") pod \"e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") " pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:48.988678 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.988357    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-envoy\") pod \"e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") " pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:48.988678 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.988412    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-workload-socket\") pod \"e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") " pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:48.988678 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.988492    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-workload-certs\") pod \"e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") " pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:48.988678 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.988546    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-credential-socket\") pod \"e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") " pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:48.989307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.989153    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-credential-socket\") pod \"e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") " pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:48.989918 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.989889    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-data\") pod \"e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") " pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:48.990937 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.990906    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istiod-ca-cert\") pod \"e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") " pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:48.991297 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.991274    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-workload-socket\") pod \"e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") " pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:48.991538 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.991515    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-workload-certs\") pod \"e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") " pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:48.994313 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.993724    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-envoy\") pod \"e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") " pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:48.995171 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:48.995139    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-podinfo\") pod \"e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") " pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:49.016617 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:49.016540    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdnbh\" (UniqueName: \"kubernetes.io/projected/1aa5f50b-7768-46fd-9ce4-76c47268bce2-kube-api-access-vdnbh\") pod \"e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") " pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:49.016617 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:49.016575    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-token\") pod \"e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") " pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:49.105754 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:49.105716    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:49.204550 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:49.204516    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:49.384463 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:49.384365    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:49.466250 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:49.454523    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"]
Oct 30 22:58:49.492816 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:58:49.492779    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa5f50b_7768_46fd_9ce4_76c47268bce2.slice/crio-f67d0e3132f28452bba8077e9aad6f603c8e9bebb113e18e435877f22259efa0 WatchSource:0}: Error finding container f67d0e3132f28452bba8077e9aad6f603c8e9bebb113e18e435877f22259efa0: Status 404 returned error can't find the container with id f67d0e3132f28452bba8077e9aad6f603c8e9bebb113e18e435877f22259efa0
Oct 30 22:58:49.497192 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:49.497148    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:58:49.497354 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:49.497313    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:58:49.497422 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:49.497364    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:58:49.563274 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:49.563203    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:49.981014 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:49.980958    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2" event={"ID":"1aa5f50b-7768-46fd-9ce4-76c47268bce2","Type":"ContainerStarted","Data":"60c45600e6986249c58dfc88800b63f8a6711b0db3111f5f480ece96a3b8be40"}
Oct 30 22:58:49.981764 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:49.981735    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2" event={"ID":"1aa5f50b-7768-46fd-9ce4-76c47268bce2","Type":"ContainerStarted","Data":"f67d0e3132f28452bba8077e9aad6f603c8e9bebb113e18e435877f22259efa0"}
Oct 30 22:58:50.057409 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:50.057319    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2" podStartSLOduration=2.057297542 podStartE2EDuration="2.057297542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:58:50.05187611 +0000 UTC m=+3115.408242428" watchObservedRunningTime="2026-10-30 22:58:50.057297542 +0000 UTC m=+3115.413663859"
Oct 30 22:58:50.106323 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:50.106287    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:50.108331 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:50.108301    4889 patch_prober.go:29] interesting pod/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Oct 30 22:58:50.108489 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:50.108360    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2" podUID="1aa5f50b-7768-46fd-9ce4-76c47268bce2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Oct 30 22:58:50.744402 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:50.744364    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:51.111293 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:51.111137    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:51.113081 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:51.113058    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:51.506383 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:51.506346    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz" podUID="d19c99b8-8347-4314-83f8-7cb17634b961" containerName="main" containerID="cri-o://e578897eec8571c8fc848017af675c7c6e61b36b70e16cce47b79750f93d5fdf" gracePeriod=45
Oct 30 22:58:51.533134 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:51.533090    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz" podUID="d19c99b8-8347-4314-83f8-7cb17634b961" containerName="main" probeResult="failure" output="Get \"https://10.132.0.225:8000/ready\": dial tcp 10.132.0.225:8000: connect: connection refused"
Oct 30 22:58:51.816765 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:51.816729    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"
Oct 30 22:58:51.956368 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:51.956334    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-model-cache\") pod \"d19c99b8-8347-4314-83f8-7cb17634b961\" (UID: \"d19c99b8-8347-4314-83f8-7cb17634b961\") "
Oct 30 22:58:51.956674 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:51.956387    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d19c99b8-8347-4314-83f8-7cb17634b961-kube-api-access-cqvxs\" (UniqueName: \"kubernetes.io/projected/d19c99b8-8347-4314-83f8-7cb17634b961-kube-api-access-cqvxs\") pod \"d19c99b8-8347-4314-83f8-7cb17634b961\" (UID: \"d19c99b8-8347-4314-83f8-7cb17634b961\") "
Oct 30 22:58:51.956674 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:51.956419    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-dshm\") pod \"d19c99b8-8347-4314-83f8-7cb17634b961\" (UID: \"d19c99b8-8347-4314-83f8-7cb17634b961\") "
Oct 30 22:58:51.956674 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:51.956480    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-home\" (UniqueName: \"kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-home\") pod \"d19c99b8-8347-4314-83f8-7cb17634b961\" (UID: \"d19c99b8-8347-4314-83f8-7cb17634b961\") "
Oct 30 22:58:51.956674 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:51.956529    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d19c99b8-8347-4314-83f8-7cb17634b961-tls-certs\" (UniqueName: \"kubernetes.io/secret/d19c99b8-8347-4314-83f8-7cb17634b961-tls-certs\") pod \"d19c99b8-8347-4314-83f8-7cb17634b961\" (UID: \"d19c99b8-8347-4314-83f8-7cb17634b961\") "
Oct 30 22:58:51.956674 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:51.956619    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-tmp-dir\") pod \"d19c99b8-8347-4314-83f8-7cb17634b961\" (UID: \"d19c99b8-8347-4314-83f8-7cb17634b961\") "
Oct 30 22:58:51.957029 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:51.956671    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-model-cache" pod "d19c99b8-8347-4314-83f8-7cb17634b961" (UID: "d19c99b8-8347-4314-83f8-7cb17634b961"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:51.957029 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:51.956981    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-tmp-dir" pod "d19c99b8-8347-4314-83f8-7cb17634b961" (UID: "d19c99b8-8347-4314-83f8-7cb17634b961"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:51.957289 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:51.957265    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-home" pod "d19c99b8-8347-4314-83f8-7cb17634b961" (UID: "d19c99b8-8347-4314-83f8-7cb17634b961"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:51.957643 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:51.957623    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:51.957716 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:51.957648    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:51.957716 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:51.957662    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:51.959363 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:51.959337    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d19c99b8-8347-4314-83f8-7cb17634b961-tls-certs" pod "d19c99b8-8347-4314-83f8-7cb17634b961" (UID: "d19c99b8-8347-4314-83f8-7cb17634b961"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:58:51.959464 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:51.959447    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-dshm" pod "d19c99b8-8347-4314-83f8-7cb17634b961" (UID: "d19c99b8-8347-4314-83f8-7cb17634b961"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:51.959528 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:51.959493    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d19c99b8-8347-4314-83f8-7cb17634b961-kube-api-access-cqvxs" pod "d19c99b8-8347-4314-83f8-7cb17634b961" (UID: "d19c99b8-8347-4314-83f8-7cb17634b961"). InnerVolumeSpecName "kube-api-access-cqvxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:58:52.001931 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:52.001898    4889 generic.go:356] "Generic (PLEG): container finished" podID="d19c99b8-8347-4314-83f8-7cb17634b961" containerID="e578897eec8571c8fc848017af675c7c6e61b36b70e16cce47b79750f93d5fdf" exitCode=0
Oct 30 22:58:52.002187 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:52.001950    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"
Oct 30 22:58:52.002187 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:52.001984    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz" event={"ID":"d19c99b8-8347-4314-83f8-7cb17634b961","Type":"ContainerDied","Data":"e578897eec8571c8fc848017af675c7c6e61b36b70e16cce47b79750f93d5fdf"}
Oct 30 22:58:52.002187 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:52.002034    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz" event={"ID":"d19c99b8-8347-4314-83f8-7cb17634b961","Type":"ContainerDied","Data":"be5d31bde5e1047b2140d9f503d0cc9e2739483235da5fc6d04dbd871e198bbc"}
Oct 30 22:58:52.002187 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:52.002056    4889 scope.go:122] "RemoveContainer" containerID="e578897eec8571c8fc848017af675c7c6e61b36b70e16cce47b79750f93d5fdf"
Oct 30 22:58:52.002559 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:52.002399    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:52.003582 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:52.003518    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 22:58:52.019020 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:52.018995    4889 scope.go:122] "RemoveContainer" containerID="e578897eec8571c8fc848017af675c7c6e61b36b70e16cce47b79750f93d5fdf"
Oct 30 22:58:52.019411 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:58:52.019385    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e578897eec8571c8fc848017af675c7c6e61b36b70e16cce47b79750f93d5fdf\": container with ID starting with e578897eec8571c8fc848017af675c7c6e61b36b70e16cce47b79750f93d5fdf not found: ID does not exist" containerID="e578897eec8571c8fc848017af675c7c6e61b36b70e16cce47b79750f93d5fdf"
Oct 30 22:58:52.019492 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:52.019420    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e578897eec8571c8fc848017af675c7c6e61b36b70e16cce47b79750f93d5fdf"} err="failed to get container status \"e578897eec8571c8fc848017af675c7c6e61b36b70e16cce47b79750f93d5fdf\": rpc error: code = NotFound desc = could not find container \"e578897eec8571c8fc848017af675c7c6e61b36b70e16cce47b79750f93d5fdf\": container with ID starting with e578897eec8571c8fc848017af675c7c6e61b36b70e16cce47b79750f93d5fdf not found: ID does not exist"
Oct 30 22:58:52.059402 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:52.059307    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqvxs\" (UniqueName: \"kubernetes.io/projected/d19c99b8-8347-4314-83f8-7cb17634b961-kube-api-access-cqvxs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:52.059402 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:52.059340    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d19c99b8-8347-4314-83f8-7cb17634b961-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:52.059402 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:52.059350    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d19c99b8-8347-4314-83f8-7cb17634b961-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:58:52.066480 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:52.066405    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"]
Oct 30 22:58:52.071976 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:52.071942    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cdc122/rate-test-model-cdc122-kserve-d5766965-p6tnz"]
Oct 30 22:58:52.611321 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:52.611281    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:52.816809 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:52.816773    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d19c99b8-8347-4314-83f8-7cb17634b961" path="/var/lib/kubelet/pods/d19c99b8-8347-4314-83f8-7cb17634b961/volumes"
Oct 30 22:58:52.832281 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:52.829011    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:53.029561 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.029521    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:53.332276 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.332138    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:53.707759 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.707650    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"]
Oct 30 22:58:53.709061 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.709030    4889 cpu_manager.go:414] "containerMap: removing container" podUID="d19c99b8-8347-4314-83f8-7cb17634b961" containerName="main"
Oct 30 22:58:53.709061 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.709061    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d19c99b8-8347-4314-83f8-7cb17634b961" containerName="main"
Oct 30 22:58:53.709377 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.709322    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="d19c99b8-8347-4314-83f8-7cb17634b961" containerName="main"
Oct 30 22:58:53.715499 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.715469    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:53.734575 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.734548    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f94561-openshift-default-dockercfg-m8xh7\""
Oct 30 22:58:53.756384 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.756343    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"]
Oct 30 22:58:53.797347 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.797303    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4966ae-49e6-461e-b677-6111da780e93-istiod-ca-cert\") pod \"e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") " pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:53.797595 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.797464    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df4966ae-49e6-461e-b677-6111da780e93-istio-token\") pod \"e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") " pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:53.797595 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.797512    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-workload-socket\") pod \"e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") " pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:53.797595 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.797548    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6s6s\" (UniqueName: \"kubernetes.io/projected/df4966ae-49e6-461e-b677-6111da780e93-kube-api-access-m6s6s\") pod \"e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") " pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:53.797809 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.797670    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-workload-certs\") pod \"e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") " pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:53.797809 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.797706    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-istio-envoy\") pod \"e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") " pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:53.797809 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.797737    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-istio-data\") pod \"e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") " pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:53.798009 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.797849    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-credential-socket\") pod \"e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") " pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:53.798009 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.797893    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4966ae-49e6-461e-b677-6111da780e93-istio-podinfo\") pod \"e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") " pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:53.808318 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.808216    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:53.906292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.901306    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4966ae-49e6-461e-b677-6111da780e93-istiod-ca-cert\") pod \"e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") " pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:53.906292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.901409    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df4966ae-49e6-461e-b677-6111da780e93-istio-token\") pod \"e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") " pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:53.906292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.901455    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-workload-socket\") pod \"e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") " pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:53.906292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.901488    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6s6s\" (UniqueName: \"kubernetes.io/projected/df4966ae-49e6-461e-b677-6111da780e93-kube-api-access-m6s6s\") pod \"e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") " pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:53.906292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.901543    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-workload-certs\") pod \"e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") " pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:53.906292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.901571    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-istio-envoy\") pod \"e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") " pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:53.906292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.901607    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-istio-data\") pod \"e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") " pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:53.906292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.901733    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-credential-socket\") pod \"e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") " pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:53.906292 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.901808    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4966ae-49e6-461e-b677-6111da780e93-istio-podinfo\") pod \"e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") " pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:53.909033 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.908865    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-workload-socket\") pod \"e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") " pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:53.909244 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.909174    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-workload-certs\") pod \"e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") " pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:53.910282 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.909473    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-istio-data\") pod \"e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") " pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:53.910910 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.910887    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-credential-socket\") pod \"e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") " pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:53.911499 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.911478    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4966ae-49e6-461e-b677-6111da780e93-istiod-ca-cert\") pod \"e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") " pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:53.913859 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.913831    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-istio-envoy\") pod \"e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") " pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:53.924278 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.918819    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4966ae-49e6-461e-b677-6111da780e93-istio-podinfo\") pod \"e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") " pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:53.935647 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.935612    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df4966ae-49e6-461e-b677-6111da780e93-istio-token\") pod \"e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") " pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:53.936235 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:53.935857    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6s6s\" (UniqueName: \"kubernetes.io/projected/df4966ae-49e6-461e-b677-6111da780e93-kube-api-access-m6s6s\") pod \"e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") " pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:54.031054 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.031024    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:54.155306 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.155217    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:54.366451 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:58:54.366416    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf4966ae_49e6_461e_b677_6111da780e93.slice/crio-d3ca3be2017c46576d1c251a61638a4fcebb3e5eb5d87f44e21a7e3e814940ee WatchSource:0}: Error finding container d3ca3be2017c46576d1c251a61638a4fcebb3e5eb5d87f44e21a7e3e814940ee: Status 404 returned error can't find the container with id d3ca3be2017c46576d1c251a61638a4fcebb3e5eb5d87f44e21a7e3e814940ee
Oct 30 22:58:54.366776 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.366752    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"]
Oct 30 22:58:54.369538 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.369500    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:58:54.369669 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.369577    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:58:54.369669 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.369607    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 22:58:54.567092 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.567057    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:54.618741 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.618652    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9"]
Oct 30 22:58:54.623257 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.623214    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9"
Oct 30 22:58:54.647361 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.647319    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-76d54a\""
Oct 30 22:58:54.650481 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.650448    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9"]
Oct 30 22:58:54.701209 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.701167    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c"]
Oct 30 22:58:54.707167 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.707141    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c"
Oct 30 22:58:54.714539 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.714507    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8af940-2412-4e39-a599-450debc2e114-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9\" (UID: \"cb8af940-2412-4e39-a599-450debc2e114\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9"
Oct 30 22:58:54.715040 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.714580    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qrjq\" (UniqueName: \"kubernetes.io/projected/cb8af940-2412-4e39-a599-450debc2e114-kube-api-access-5qrjq\") pod \"payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9\" (UID: \"cb8af940-2412-4e39-a599-450debc2e114\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9"
Oct 30 22:58:54.725905 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.725872    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-76d54a-dockercfg-lf5k6\""
Oct 30 22:58:54.730444 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.730411    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c"]
Oct 30 22:58:54.858070 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.856257    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgml9\" (UniqueName: \"kubernetes.io/projected/b1ea4fda-c921-46a1-8107-441bea92acf1-kube-api-access-fgml9\") pod \"payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c\" (UID: \"b1ea4fda-c921-46a1-8107-441bea92acf1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c"
Oct 30 22:58:54.858070 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.856495    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8af940-2412-4e39-a599-450debc2e114-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9\" (UID: \"cb8af940-2412-4e39-a599-450debc2e114\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9"
Oct 30 22:58:54.858070 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.856571    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qrjq\" (UniqueName: \"kubernetes.io/projected/cb8af940-2412-4e39-a599-450debc2e114-kube-api-access-5qrjq\") pod \"payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9\" (UID: \"cb8af940-2412-4e39-a599-450debc2e114\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9"
Oct 30 22:58:54.858070 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.856599    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ea4fda-c921-46a1-8107-441bea92acf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c\" (UID: \"b1ea4fda-c921-46a1-8107-441bea92acf1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c"
Oct 30 22:58:54.895401 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.880563    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-76d54a\""
Oct 30 22:58:54.895401 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.882544    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:54.895401 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.884941    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qrjq\" (UniqueName: \"kubernetes.io/projected/cb8af940-2412-4e39-a599-450debc2e114-kube-api-access-5qrjq\") pod \"payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9\" (UID: \"cb8af940-2412-4e39-a599-450debc2e114\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9"
Oct 30 22:58:54.895401 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.889701    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8af940-2412-4e39-a599-450debc2e114-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9\" (UID: \"cb8af940-2412-4e39-a599-450debc2e114\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9"
Oct 30 22:58:54.942239 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.942181    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9"
Oct 30 22:58:54.957895 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.957850    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ea4fda-c921-46a1-8107-441bea92acf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c\" (UID: \"b1ea4fda-c921-46a1-8107-441bea92acf1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c"
Oct 30 22:58:54.958106 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.957991    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgml9\" (UniqueName: \"kubernetes.io/projected/b1ea4fda-c921-46a1-8107-441bea92acf1-kube-api-access-fgml9\") pod \"payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c\" (UID: \"b1ea4fda-c921-46a1-8107-441bea92acf1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c"
Oct 30 22:58:54.958680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.958651    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ea4fda-c921-46a1-8107-441bea92acf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c\" (UID: \"b1ea4fda-c921-46a1-8107-441bea92acf1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c"
Oct 30 22:58:54.986097 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:54.986045    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgml9\" (UniqueName: \"kubernetes.io/projected/b1ea4fda-c921-46a1-8107-441bea92acf1-kube-api-access-fgml9\") pod \"payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c\" (UID: \"b1ea4fda-c921-46a1-8107-441bea92acf1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c"
Oct 30 22:58:55.047754 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:55.047710    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-76d54a-dockercfg-lf5k6\""
Oct 30 22:58:55.049134 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:55.049108    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c"
Oct 30 22:58:55.054376 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:55.054342    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh" event={"ID":"df4966ae-49e6-461e-b677-6111da780e93","Type":"ContainerStarted","Data":"2f9d5c5f3c357c6ed6f1e5fe72ee40e13646424e2f2b8dd53d6cb0e392d8d665"}
Oct 30 22:58:55.054947 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:55.054602    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh" event={"ID":"df4966ae-49e6-461e-b677-6111da780e93","Type":"ContainerStarted","Data":"d3ca3be2017c46576d1c251a61638a4fcebb3e5eb5d87f44e21a7e3e814940ee"}
Oct 30 22:58:55.284674 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:55.284609    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh" podStartSLOduration=2.284593011 podStartE2EDuration="2.284593011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:58:55.123257883 +0000 UTC m=+3120.479624217" watchObservedRunningTime="2026-10-30 22:58:55.284593011 +0000 UTC m=+3120.640959326"
Oct 30 22:58:55.285450 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:58:55.285414    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8af940_2412_4e39_a599_450debc2e114.slice/crio-3f507f15175e1deb95b1c8434218a4c59582528b2b011b1270bd56f09a143b46 WatchSource:0}: Error finding container 3f507f15175e1deb95b1c8434218a4c59582528b2b011b1270bd56f09a143b46: Status 404 returned error can't find the container with id 3f507f15175e1deb95b1c8434218a4c59582528b2b011b1270bd56f09a143b46
Oct 30 22:58:55.285676 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:55.285659    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9"]
Oct 30 22:58:55.377145 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:55.377112    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:55.379439 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:55.379338    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c"]
Oct 30 22:58:55.656107 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:55.656009    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:56.031963 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:56.031930    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:56.039054 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:56.039021    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:56.065702 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:56.065660    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9" event={"ID":"cb8af940-2412-4e39-a599-450debc2e114","Type":"ContainerStarted","Data":"a5d31ac981a7a9c5c3603cf1c6bfbb0ad9cdce1a09b27c9f64270c9688916d50"}
Oct 30 22:58:56.065914 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:56.065720    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9" event={"ID":"cb8af940-2412-4e39-a599-450debc2e114","Type":"ContainerStarted","Data":"3f507f15175e1deb95b1c8434218a4c59582528b2b011b1270bd56f09a143b46"}
Oct 30 22:58:56.065914 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:56.065751    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9"
Oct 30 22:58:56.067471 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:56.067443    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c" event={"ID":"b1ea4fda-c921-46a1-8107-441bea92acf1","Type":"ContainerStarted","Data":"0ea9d4bef8cba1bd9de15e7344e5aaca13c4be5244f3f7c4b79ec50c5f447a62"}
Oct 30 22:58:56.067621 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:56.067488    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c" event={"ID":"b1ea4fda-c921-46a1-8107-441bea92acf1","Type":"ContainerStarted","Data":"668ac023fb4f4adcbcd93f437143aed61c899002bcba20163dcd8c8af247edcf"}
Oct 30 22:58:56.067621 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:56.067536    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c"
Oct 30 22:58:56.067768 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:56.067750    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:56.068898 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:56.068880    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 22:58:56.199282 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:56.199190    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c" podStartSLOduration=2.199175622 podStartE2EDuration="2.199175622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:58:56.194428653 +0000 UTC m=+3121.550794969" watchObservedRunningTime="2026-10-30 22:58:56.199175622 +0000 UTC m=+3121.555541937"
Oct 30 22:58:57.290200 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:57.290141    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:57.625430 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:57.625385    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:57.824059 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:57.824022    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:59.418052 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:59.418021    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:59.469343 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:59.469267    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9" podStartSLOduration=5.469249617 podStartE2EDuration="5.469249617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:58:56.241397334 +0000 UTC m=+3121.597763650" watchObservedRunningTime="2026-10-30 22:58:59.469249617 +0000 UTC m=+3124.825615932"
Oct 30 22:58:59.470302 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:59.470283    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt"]
Oct 30 22:58:59.476622 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:59.476599    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt"
Oct 30 22:58:59.495659 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:59.495629    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt"]
Oct 30 22:58:59.504429 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:59.504386    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f94561\""
Oct 30 22:58:59.532172 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:59.532112    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t6g8\" (UniqueName: \"kubernetes.io/projected/9939f760-86ef-484a-876c-6495d528af48-kube-api-access-8t6g8\") pod \"payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt\" (UID: \"9939f760-86ef-484a-876c-6495d528af48\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt"
Oct 30 22:58:59.532415 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:59.532309    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9939f760-86ef-484a-876c-6495d528af48-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt\" (UID: \"9939f760-86ef-484a-876c-6495d528af48\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt"
Oct 30 22:58:59.632290 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:59.619200    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:59.656357 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:59.635766    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t6g8\" (UniqueName: \"kubernetes.io/projected/9939f760-86ef-484a-876c-6495d528af48-kube-api-access-8t6g8\") pod \"payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt\" (UID: \"9939f760-86ef-484a-876c-6495d528af48\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt"
Oct 30 22:58:59.656357 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:59.636434    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9939f760-86ef-484a-876c-6495d528af48-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt\" (UID: \"9939f760-86ef-484a-876c-6495d528af48\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt"
Oct 30 22:58:59.656357 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:59.638275    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9939f760-86ef-484a-876c-6495d528af48-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt\" (UID: \"9939f760-86ef-484a-876c-6495d528af48\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt"
Oct 30 22:58:59.712304 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:59.700057    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t6g8\" (UniqueName: \"kubernetes.io/projected/9939f760-86ef-484a-876c-6495d528af48-kube-api-access-8t6g8\") pod \"payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt\" (UID: \"9939f760-86ef-484a-876c-6495d528af48\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt"
Oct 30 22:58:59.795717 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:58:59.795675    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt"
Oct 30 22:59:00.131558 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:59:00.131525    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9939f760_86ef_484a_876c_6495d528af48.slice/crio-d4f48759f5813d97416f2b27c98f757a6bca16ac94443031164ad4be4cf75b07 WatchSource:0}: Error finding container d4f48759f5813d97416f2b27c98f757a6bca16ac94443031164ad4be4cf75b07: Status 404 returned error can't find the container with id d4f48759f5813d97416f2b27c98f757a6bca16ac94443031164ad4be4cf75b07
Oct 30 22:59:00.132920 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:00.132893    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt"]
Oct 30 22:59:01.110685 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:01.110652    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt" event={"ID":"9939f760-86ef-484a-876c-6495d528af48","Type":"ContainerStarted","Data":"47ebd65232f113b89686bc9294e80f7ef13c592c276d9bf548eb601ec244bb34"}
Oct 30 22:59:01.111118 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:01.110704    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt" event={"ID":"9939f760-86ef-484a-876c-6495d528af48","Type":"ContainerStarted","Data":"d4f48759f5813d97416f2b27c98f757a6bca16ac94443031164ad4be4cf75b07"}
Oct 30 22:59:01.111118 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:01.110770    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt"
Oct 30 22:59:01.158279 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:01.158207    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt" podStartSLOduration=2.158193329 podStartE2EDuration="2.158193329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:59:01.156524419 +0000 UTC m=+3126.512890736" watchObservedRunningTime="2026-10-30 22:59:01.158193329 +0000 UTC m=+3126.514559641"
Oct 30 22:59:01.482052 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:01.481954    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:03.371565 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:03.371520    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:04.792818 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:04.792766    4889 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","besteffort","pod5f4c1b10-0d9c-4bb7-a49a-df55737c6a05"] err="unable to destroy cgroup paths for cgroup [kubepods besteffort pod5f4c1b10-0d9c-4bb7-a49a-df55737c6a05] : Timed out while waiting for systemd to remove kubepods-besteffort-pod5f4c1b10_0d9c_4bb7_a49a_df55737c6a05.slice"
Oct 30 22:59:04.792818 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:59:04.792819    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods besteffort pod5f4c1b10-0d9c-4bb7-a49a-df55737c6a05] : unable to destroy cgroup paths for cgroup [kubepods besteffort pod5f4c1b10-0d9c-4bb7-a49a-df55737c6a05] : Timed out while waiting for systemd to remove kubepods-besteffort-pod5f4c1b10_0d9c_4bb7_a49a_df55737c6a05.slice" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05"
Oct 30 22:59:05.146903 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:05.146785    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:05.152873 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:05.152833    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"
Oct 30 22:59:05.255001 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:05.254966    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"]
Oct 30 22:59:05.259422 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:05.259396    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0ad7c5/simulator-w4-0ad7c5-kserve-7b775c46b9-tqkkp"]
Oct 30 22:59:06.107810 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.107767    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"]
Oct 30 22:59:06.114531 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.114505    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"
Oct 30 22:59:06.132385 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.131571    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"]
Oct 30 22:59:06.138625 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.138587    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-23aab2\"/\"openshift-service-ca.crt\""
Oct 30 22:59:06.139821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.139018    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-23aab2\"/\"sub-test-model-23aab2-kserve-self-signed-certs\""
Oct 30 22:59:06.139821 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.139146    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-23aab2\"/\"kube-root-ca.crt\""
Oct 30 22:59:06.144437 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.144411    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-23aab2\"/\"default-dockercfg-xtg2q\""
Oct 30 22:59:06.230972 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.230932    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:06.243319 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.243280    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-home\") pod \"sub-test-model-23aab2-kserve-d584986c5-f276l\" (UID: \"7967e3f2-562e-476a-8e1e-294926499cee\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"
Oct 30 22:59:06.243549 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.243382    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhhg8\" (UniqueName: \"kubernetes.io/projected/7967e3f2-562e-476a-8e1e-294926499cee-kube-api-access-qhhg8\") pod \"sub-test-model-23aab2-kserve-d584986c5-f276l\" (UID: \"7967e3f2-562e-476a-8e1e-294926499cee\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"
Oct 30 22:59:06.243549 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.243442    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-tmp-dir\") pod \"sub-test-model-23aab2-kserve-d584986c5-f276l\" (UID: \"7967e3f2-562e-476a-8e1e-294926499cee\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"
Oct 30 22:59:06.243676 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.243551    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-model-cache\") pod \"sub-test-model-23aab2-kserve-d584986c5-f276l\" (UID: \"7967e3f2-562e-476a-8e1e-294926499cee\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"
Oct 30 22:59:06.243676 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.243627    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7967e3f2-562e-476a-8e1e-294926499cee-tls-certs\") pod \"sub-test-model-23aab2-kserve-d584986c5-f276l\" (UID: \"7967e3f2-562e-476a-8e1e-294926499cee\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"
Oct 30 22:59:06.243791 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.243701    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-dshm\") pod \"sub-test-model-23aab2-kserve-d584986c5-f276l\" (UID: \"7967e3f2-562e-476a-8e1e-294926499cee\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"
Oct 30 22:59:06.345184 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.345145    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7967e3f2-562e-476a-8e1e-294926499cee-tls-certs\") pod \"sub-test-model-23aab2-kserve-d584986c5-f276l\" (UID: \"7967e3f2-562e-476a-8e1e-294926499cee\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"
Oct 30 22:59:06.346094 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.346067    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-dshm\") pod \"sub-test-model-23aab2-kserve-d584986c5-f276l\" (UID: \"7967e3f2-562e-476a-8e1e-294926499cee\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"
Oct 30 22:59:06.360556 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.360454    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-home\") pod \"sub-test-model-23aab2-kserve-d584986c5-f276l\" (UID: \"7967e3f2-562e-476a-8e1e-294926499cee\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"
Oct 30 22:59:06.363307 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.360821    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhhg8\" (UniqueName: \"kubernetes.io/projected/7967e3f2-562e-476a-8e1e-294926499cee-kube-api-access-qhhg8\") pod \"sub-test-model-23aab2-kserve-d584986c5-f276l\" (UID: \"7967e3f2-562e-476a-8e1e-294926499cee\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"
Oct 30 22:59:06.363680 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.363648    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-tmp-dir\") pod \"sub-test-model-23aab2-kserve-d584986c5-f276l\" (UID: \"7967e3f2-562e-476a-8e1e-294926499cee\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"
Oct 30 22:59:06.371247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.369449    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-model-cache\") pod \"sub-test-model-23aab2-kserve-d584986c5-f276l\" (UID: \"7967e3f2-562e-476a-8e1e-294926499cee\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"
Oct 30 22:59:06.371247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.370148    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-model-cache\") pod \"sub-test-model-23aab2-kserve-d584986c5-f276l\" (UID: \"7967e3f2-562e-476a-8e1e-294926499cee\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"
Oct 30 22:59:06.371247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.370437    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-home\") pod \"sub-test-model-23aab2-kserve-d584986c5-f276l\" (UID: \"7967e3f2-562e-476a-8e1e-294926499cee\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"
Oct 30 22:59:06.371247 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.371010    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-tmp-dir\") pod \"sub-test-model-23aab2-kserve-d584986c5-f276l\" (UID: \"7967e3f2-562e-476a-8e1e-294926499cee\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"
Oct 30 22:59:06.393311 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.392118    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7967e3f2-562e-476a-8e1e-294926499cee-tls-certs\") pod \"sub-test-model-23aab2-kserve-d584986c5-f276l\" (UID: \"7967e3f2-562e-476a-8e1e-294926499cee\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"
Oct 30 22:59:06.402303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.400081    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhhg8\" (UniqueName: \"kubernetes.io/projected/7967e3f2-562e-476a-8e1e-294926499cee-kube-api-access-qhhg8\") pod \"sub-test-model-23aab2-kserve-d584986c5-f276l\" (UID: \"7967e3f2-562e-476a-8e1e-294926499cee\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"
Oct 30 22:59:06.402303 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.400530    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-dshm\") pod \"sub-test-model-23aab2-kserve-d584986c5-f276l\" (UID: \"7967e3f2-562e-476a-8e1e-294926499cee\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"
Oct 30 22:59:06.435145 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.435097    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"
Oct 30 22:59:06.632049 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.631946    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:06.781330 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.781296    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f4c1b10-0d9c-4bb7-a49a-df55737c6a05" path="/var/lib/kubelet/pods/5f4c1b10-0d9c-4bb7-a49a-df55737c6a05/volumes"
Oct 30 22:59:06.781768 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:06.781746    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"]
Oct 30 22:59:07.004541 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:07.004505    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:07.079853 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:07.079825    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9"
Oct 30 22:59:07.080060 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:07.079908    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c"
Oct 30 22:59:07.209259 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:07.205884    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l" event={"ID":"7967e3f2-562e-476a-8e1e-294926499cee","Type":"ContainerStarted","Data":"4ff87db5ab76260f525f3e4b56e1c424a067f4665ed892424bc0fc6faefc81da"}
Oct 30 22:59:07.209259 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:07.205937    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l" event={"ID":"7967e3f2-562e-476a-8e1e-294926499cee","Type":"ContainerStarted","Data":"d462433fb4035d95e2f38eee6ce4dda779cfc10ff4df3ab105ff44600c797f12"}
Oct 30 22:59:07.370749 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:07.370632    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:07.414038 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:07.413955    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l" podStartSLOduration=1.413938605 podStartE2EDuration="1.413938605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:59:07.410593411 +0000 UTC m=+3132.766959725" watchObservedRunningTime="2026-10-30 22:59:07.413938605 +0000 UTC m=+3132.770304920"
Oct 30 22:59:07.435485 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:07.435449    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"
Oct 30 22:59:07.540715 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:07.540674    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:08.416368 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:08.416314    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:08.589066 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:08.589020    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:09.069305 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:09.069256    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:10.108716 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:10.108684    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:10.306207 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:59:10.306170    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:59:10.959348 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:10.959313    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:11.140122 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:11.140087    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:12.123892 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:12.123860    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt"
Oct 30 22:59:12.689687 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:12.689647    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:12.776887 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:12.776851    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp"]
Oct 30 22:59:12.784647 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:12.784607    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp"
Oct 30 22:59:12.802004 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:12.801969    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp"]
Oct 30 22:59:12.804657 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:12.804616    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-cdfa86-dockercfg-7shml\""
Oct 30 22:59:12.805355 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:12.805317    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-cdfa86\""
Oct 30 22:59:12.817016 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:12.816978    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrlwq\" (UniqueName: \"kubernetes.io/projected/3e35e8fe-0081-41ff-851f-c61c25c3942a-kube-api-access-qrlwq\") pod \"payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp\" (UID: \"3e35e8fe-0081-41ff-851f-c61c25c3942a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp"
Oct 30 22:59:12.817212 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:12.817117    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e35e8fe-0081-41ff-851f-c61c25c3942a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp\" (UID: \"3e35e8fe-0081-41ff-851f-c61c25c3942a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp"
Oct 30 22:59:12.872892 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:12.872850    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:12.932013 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:12.931947    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e35e8fe-0081-41ff-851f-c61c25c3942a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp\" (UID: \"3e35e8fe-0081-41ff-851f-c61c25c3942a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp"
Oct 30 22:59:12.948885 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:12.947509    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e35e8fe-0081-41ff-851f-c61c25c3942a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp\" (UID: \"3e35e8fe-0081-41ff-851f-c61c25c3942a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp"
Oct 30 22:59:12.962578 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:12.962533    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrlwq\" (UniqueName: \"kubernetes.io/projected/3e35e8fe-0081-41ff-851f-c61c25c3942a-kube-api-access-qrlwq\") pod \"payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp\" (UID: \"3e35e8fe-0081-41ff-851f-c61c25c3942a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp"
Oct 30 22:59:13.007041 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:13.007001    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrlwq\" (UniqueName: \"kubernetes.io/projected/3e35e8fe-0081-41ff-851f-c61c25c3942a-kube-api-access-qrlwq\") pod \"payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp\" (UID: \"3e35e8fe-0081-41ff-851f-c61c25c3942a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp"
Oct 30 22:59:13.104444 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:13.104405    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp"
Oct 30 22:59:13.402621 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:13.402577    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:13.421087 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:59:13.421052    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e35e8fe_0081_41ff_851f_c61c25c3942a.slice/crio-7a17e7123c6801156728d6af1f355e50ea977c621622a146958d1896611ffd56 WatchSource:0}: Error finding container 7a17e7123c6801156728d6af1f355e50ea977c621622a146958d1896611ffd56: Status 404 returned error can't find the container with id 7a17e7123c6801156728d6af1f355e50ea977c621622a146958d1896611ffd56
Oct 30 22:59:13.421409 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:13.421369    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp"]
Oct 30 22:59:14.273389 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:14.273344    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp" event={"ID":"3e35e8fe-0081-41ff-851f-c61c25c3942a","Type":"ContainerStarted","Data":"74bd7031b4b5c117bfd0bf7f80654127e56552c6bbb3df532e280c2d201595fb"}
Oct 30 22:59:14.273910 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:14.273424    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp" event={"ID":"3e35e8fe-0081-41ff-851f-c61c25c3942a","Type":"ContainerStarted","Data":"7a17e7123c6801156728d6af1f355e50ea977c621622a146958d1896611ffd56"}
Oct 30 22:59:14.273910 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:14.273456    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp"
Oct 30 22:59:14.319095 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:14.319037    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp" podStartSLOduration=2.319021622 podStartE2EDuration="2.319021622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:59:14.317950866 +0000 UTC m=+3139.674317183" watchObservedRunningTime="2026-10-30 22:59:14.319021622 +0000 UTC m=+3139.675387936"
Oct 30 22:59:15.338936 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:15.338863    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:16.435835 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:16.435796    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"
Oct 30 22:59:16.449442 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:16.449407    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"
Oct 30 22:59:16.738864 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:16.738757    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:17.187700 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:17.187658    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:17.308203 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:17.308171    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"
Oct 30 22:59:19.430008 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:19.429966    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:21.099510 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:21.099316    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:22.909496 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:22.909296    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:23.929551 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:23.929511    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:24.725138 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:24.725097    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:25.285281 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:25.285248    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp"
Oct 30 22:59:26.431622 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:26.431575    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:28.374036 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:28.373997    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:30.344999 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:30.344426    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:31.324701 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:31.324666    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:31.528333 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:31.528281    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:32.445244 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:32.445181    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228"]
Oct 30 22:59:32.445529 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:32.445479    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228" podUID="aa389a05-73a7-490b-8d2d-a8e34d82b0af" containerName="maas-api" containerID="cri-o://0a9012f3ee7a69fa2936a425036de23ab089c90d6eb964df79f466ed036bdd93" gracePeriod=30
Oct 30 22:59:32.658641 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:32.658453    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:32.793113 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:32.793082    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228"
Oct 30 22:59:32.905034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:32.904578    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa389a05-73a7-490b-8d2d-a8e34d82b0af-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa389a05-73a7-490b-8d2d-a8e34d82b0af-maas-api-tls\") pod \"aa389a05-73a7-490b-8d2d-a8e34d82b0af\" (UID: \"aa389a05-73a7-490b-8d2d-a8e34d82b0af\") "
Oct 30 22:59:32.905034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:32.904732    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa389a05-73a7-490b-8d2d-a8e34d82b0af-kube-api-access-79gb4\" (UniqueName: \"kubernetes.io/projected/aa389a05-73a7-490b-8d2d-a8e34d82b0af-kube-api-access-79gb4\") pod \"aa389a05-73a7-490b-8d2d-a8e34d82b0af\" (UID: \"aa389a05-73a7-490b-8d2d-a8e34d82b0af\") "
Oct 30 22:59:32.905034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:32.904849    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa389a05-73a7-490b-8d2d-a8e34d82b0af-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa389a05-73a7-490b-8d2d-a8e34d82b0af-openshift-service-ca\") pod \"aa389a05-73a7-490b-8d2d-a8e34d82b0af\" (UID: \"aa389a05-73a7-490b-8d2d-a8e34d82b0af\") "
Oct 30 22:59:32.906767 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:32.905555    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa389a05-73a7-490b-8d2d-a8e34d82b0af-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa389a05-73a7-490b-8d2d-a8e34d82b0af-metrics-certs\") pod \"aa389a05-73a7-490b-8d2d-a8e34d82b0af\" (UID: \"aa389a05-73a7-490b-8d2d-a8e34d82b0af\") "
Oct 30 22:59:32.907547 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:32.907498    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa389a05-73a7-490b-8d2d-a8e34d82b0af-openshift-service-ca" pod "aa389a05-73a7-490b-8d2d-a8e34d82b0af" (UID: "aa389a05-73a7-490b-8d2d-a8e34d82b0af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:59:32.973258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:32.965572    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa389a05-73a7-490b-8d2d-a8e34d82b0af-metrics-certs" pod "aa389a05-73a7-490b-8d2d-a8e34d82b0af" (UID: "aa389a05-73a7-490b-8d2d-a8e34d82b0af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:59:32.973258 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:32.972508    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa389a05-73a7-490b-8d2d-a8e34d82b0af-kube-api-access-79gb4" pod "aa389a05-73a7-490b-8d2d-a8e34d82b0af" (UID: "aa389a05-73a7-490b-8d2d-a8e34d82b0af"). InnerVolumeSpecName "kube-api-access-79gb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:59:32.974170 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:32.974041    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa389a05-73a7-490b-8d2d-a8e34d82b0af-maas-api-tls" pod "aa389a05-73a7-490b-8d2d-a8e34d82b0af" (UID: "aa389a05-73a7-490b-8d2d-a8e34d82b0af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:59:33.012954 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:33.012835    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79gb4\" (UniqueName: \"kubernetes.io/projected/aa389a05-73a7-490b-8d2d-a8e34d82b0af-kube-api-access-79gb4\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:59:33.012954 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:33.012877    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa389a05-73a7-490b-8d2d-a8e34d82b0af-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:59:33.012954 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:33.012893    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa389a05-73a7-490b-8d2d-a8e34d82b0af-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:59:33.012954 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:33.012908    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa389a05-73a7-490b-8d2d-a8e34d82b0af-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 22:59:33.232686 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:33.232571    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:33.421034 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:33.420989    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:33.460417 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:33.460378    4889 generic.go:356] "Generic (PLEG): container finished" podID="aa389a05-73a7-490b-8d2d-a8e34d82b0af" containerID="0a9012f3ee7a69fa2936a425036de23ab089c90d6eb964df79f466ed036bdd93" exitCode=0
Oct 30 22:59:33.460665 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:33.460600    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228" event={"ID":"aa389a05-73a7-490b-8d2d-a8e34d82b0af","Type":"ContainerDied","Data":"0a9012f3ee7a69fa2936a425036de23ab089c90d6eb964df79f466ed036bdd93"}
Oct 30 22:59:33.460740 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:33.460686    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228" event={"ID":"aa389a05-73a7-490b-8d2d-a8e34d82b0af","Type":"ContainerDied","Data":"84698186aaf6328aa2cd903f0a7d4ae5a0fffcd3dfa1d44aa2b80990370e3225"}
Oct 30 22:59:33.460740 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:33.460719    4889 scope.go:122] "RemoveContainer" containerID="0a9012f3ee7a69fa2936a425036de23ab089c90d6eb964df79f466ed036bdd93"
Oct 30 22:59:33.460962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:33.460927    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228"
Oct 30 22:59:33.472942 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:33.472914    4889 scope.go:122] "RemoveContainer" containerID="0a9012f3ee7a69fa2936a425036de23ab089c90d6eb964df79f466ed036bdd93"
Oct 30 22:59:33.473308 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:59:33.473281    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a9012f3ee7a69fa2936a425036de23ab089c90d6eb964df79f466ed036bdd93\": container with ID starting with 0a9012f3ee7a69fa2936a425036de23ab089c90d6eb964df79f466ed036bdd93 not found: ID does not exist" containerID="0a9012f3ee7a69fa2936a425036de23ab089c90d6eb964df79f466ed036bdd93"
Oct 30 22:59:33.473404 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:33.473316    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9012f3ee7a69fa2936a425036de23ab089c90d6eb964df79f466ed036bdd93"} err="failed to get container status \"0a9012f3ee7a69fa2936a425036de23ab089c90d6eb964df79f466ed036bdd93\": rpc error: code = NotFound desc = could not find container \"0a9012f3ee7a69fa2936a425036de23ab089c90d6eb964df79f466ed036bdd93\": container with ID starting with 0a9012f3ee7a69fa2936a425036de23ab089c90d6eb964df79f466ed036bdd93 not found: ID does not exist"
Oct 30 22:59:33.527522 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:33.527479    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228"]
Oct 30 22:59:33.538283 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:33.538207    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0ad7c5-55588dc775-pv228"]
Oct 30 22:59:33.598777 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:33.598735    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:33.869465 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:33.869366    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:34.777679 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:34.777642    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa389a05-73a7-490b-8d2d-a8e34d82b0af" path="/var/lib/kubelet/pods/aa389a05-73a7-490b-8d2d-a8e34d82b0af/volumes"
Oct 30 22:59:36.095186 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:36.095143    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:37.798966 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:37.798931    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:37.990152 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:37.990111    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:38.165879 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:38.165789    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:39.830309 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:39.830272    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:40.305457 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:59:40.305419    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 22:59:40.822011 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:40.821924    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:40.999300 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:40.999264    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:41.228024 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:41.227925    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:41.663171 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:41.663089    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:43.263314 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:43.263264    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf"]
Oct 30 22:59:43.264248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:43.264207    4889 cpu_manager.go:414] "containerMap: removing container" podUID="aa389a05-73a7-490b-8d2d-a8e34d82b0af" containerName="maas-api"
Oct 30 22:59:43.264248 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:43.264249    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa389a05-73a7-490b-8d2d-a8e34d82b0af" containerName="maas-api"
Oct 30 22:59:43.264449 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:43.264434    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa389a05-73a7-490b-8d2d-a8e34d82b0af" containerName="maas-api"
Oct 30 22:59:43.269880 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:43.269855    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf"
Oct 30 22:59:43.280840 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:43.280807    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/664cb2dc-09fb-45ca-b315-7ab915110da7-metrics-certs\") pod \"maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf\" (UID: \"664cb2dc-09fb-45ca-b315-7ab915110da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf"
Oct 30 22:59:43.281159 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:43.280861    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n27f\" (UniqueName: \"kubernetes.io/projected/664cb2dc-09fb-45ca-b315-7ab915110da7-kube-api-access-9n27f\") pod \"maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf\" (UID: \"664cb2dc-09fb-45ca-b315-7ab915110da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf"
Oct 30 22:59:43.281159 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:43.280999    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/664cb2dc-09fb-45ca-b315-7ab915110da7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf\" (UID: \"664cb2dc-09fb-45ca-b315-7ab915110da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf"
Oct 30 22:59:43.281695 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:43.281218    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/664cb2dc-09fb-45ca-b315-7ab915110da7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf\" (UID: \"664cb2dc-09fb-45ca-b315-7ab915110da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf"
Oct 30 22:59:43.291009 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:43.290546    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f94561-serving-cert\""
Oct 30 22:59:43.295891 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:43.295850    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf"]
Oct 30 22:59:43.382027 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:43.381975    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/664cb2dc-09fb-45ca-b315-7ab915110da7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf\" (UID: \"664cb2dc-09fb-45ca-b315-7ab915110da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf"
Oct 30 22:59:43.382289 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:43.382054    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/664cb2dc-09fb-45ca-b315-7ab915110da7-metrics-certs\") pod \"maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf\" (UID: \"664cb2dc-09fb-45ca-b315-7ab915110da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf"
Oct 30 22:59:43.382289 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:43.382089    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n27f\" (UniqueName: \"kubernetes.io/projected/664cb2dc-09fb-45ca-b315-7ab915110da7-kube-api-access-9n27f\") pod \"maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf\" (UID: \"664cb2dc-09fb-45ca-b315-7ab915110da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf"
Oct 30 22:59:43.382289 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:59:43.382161    4889 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-serving-cert: secret "maas-api-e2e-ipp-a-f94561-serving-cert" not found
Oct 30 22:59:43.382289 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:43.382201    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/664cb2dc-09fb-45ca-b315-7ab915110da7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf\" (UID: \"664cb2dc-09fb-45ca-b315-7ab915110da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf"
Oct 30 22:59:43.382522 ip-10-0-142-185 kubenswrapper[4889]: E1030 22:59:43.382296    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/664cb2dc-09fb-45ca-b315-7ab915110da7-maas-api-tls podName:664cb2dc-09fb-45ca-b315-7ab915110da7 nodeName:}" failed. No retries permitted until 2026-10-30 22:59:43.882275567 +0000 UTC m=+3169.238641881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/664cb2dc-09fb-45ca-b315-7ab915110da7-maas-api-tls") pod "maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf" (UID: "664cb2dc-09fb-45ca-b315-7ab915110da7") : secret "maas-api-e2e-ipp-a-f94561-serving-cert" not found
Oct 30 22:59:43.382986 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:43.382958    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/664cb2dc-09fb-45ca-b315-7ab915110da7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf\" (UID: \"664cb2dc-09fb-45ca-b315-7ab915110da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf"
Oct 30 22:59:43.384947 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:43.384914    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/664cb2dc-09fb-45ca-b315-7ab915110da7-metrics-certs\") pod \"maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf\" (UID: \"664cb2dc-09fb-45ca-b315-7ab915110da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf"
Oct 30 22:59:43.406407 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:43.406371    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n27f\" (UniqueName: \"kubernetes.io/projected/664cb2dc-09fb-45ca-b315-7ab915110da7-kube-api-access-9n27f\") pod \"maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf\" (UID: \"664cb2dc-09fb-45ca-b315-7ab915110da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf"
Oct 30 22:59:43.424866 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:43.424812    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:43.617933 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:43.617830    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:43.890345 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:43.890209    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/664cb2dc-09fb-45ca-b315-7ab915110da7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf\" (UID: \"664cb2dc-09fb-45ca-b315-7ab915110da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf"
Oct 30 22:59:43.893030 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:43.893001    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/664cb2dc-09fb-45ca-b315-7ab915110da7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf\" (UID: \"664cb2dc-09fb-45ca-b315-7ab915110da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf"
Oct 30 22:59:43.900916 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:43.900886    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf"
Oct 30 22:59:44.222281 ip-10-0-142-185 kubenswrapper[4889]: W1030 22:59:44.222248    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod664cb2dc_09fb_45ca_b315_7ab915110da7.slice/crio-85e5be789b867f1a6eefdcce88e7f6dccd5a8810331498bb609821d75dc83d82 WatchSource:0}: Error finding container 85e5be789b867f1a6eefdcce88e7f6dccd5a8810331498bb609821d75dc83d82: Status 404 returned error can't find the container with id 85e5be789b867f1a6eefdcce88e7f6dccd5a8810331498bb609821d75dc83d82
Oct 30 22:59:44.222457 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:44.222366    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf"]
Oct 30 22:59:44.559509 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:44.559470    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf" event={"ID":"664cb2dc-09fb-45ca-b315-7ab915110da7","Type":"ContainerStarted","Data":"85e5be789b867f1a6eefdcce88e7f6dccd5a8810331498bb609821d75dc83d82"}
Oct 30 22:59:45.214125 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:45.214075    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:45.570010 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:45.569973    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf" event={"ID":"664cb2dc-09fb-45ca-b315-7ab915110da7","Type":"ContainerStarted","Data":"ac9513720d3f26a79085d38105323e9a3df48c23cf735f860b99ef51bde811e5"}
Oct 30 22:59:45.570490 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:45.570041    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf"
Oct 30 22:59:45.615567 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:45.615503    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf" podStartSLOduration=2.302883772 podStartE2EDuration="2.615486061s" totalImagesPullingTime="312.602289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:59:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:59:45.613568664 +0000 UTC m=+3170.969934980" watchObservedRunningTime="2026-10-30 22:59:45.615486061 +0000 UTC m=+3170.971852379"
Oct 30 22:59:46.959269 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:46.959191    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:48.612651 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:48.612591    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:50.332832 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:50.332777    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:51.217724 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:51.217682    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:51.591380 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:51.591348    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf"
Oct 30 22:59:52.101756 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:52.101712    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:53.841610 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:53.841559    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:54.609446 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:54.609407    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:55.640399 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:55.640350    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:57.281404 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:57.281364    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:57.461108 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:57.461072    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:59.224962 ip-10-0-142-185 kubenswrapper[4889]: I1030 22:59:59.224920    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:00.208332 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.208293    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2"]
Oct 30 23:00:00.214972 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.214932    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h"]
Oct 30 23:00:00.216009 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.215647    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2"
Oct 30 23:00:00.222813 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.222782    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h"
Oct 30 23:00:00.294794 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.294755    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2"]
Oct 30 23:00:00.305006 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.301970    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe9cad0-baac-45db-a8e4-63a6e83d00c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2\" (UID: \"4fe9cad0-baac-45db-a8e4-63a6e83d00c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2"
Oct 30 23:00:00.305006 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.302144    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfjlm\" (UniqueName: \"kubernetes.io/projected/4fe9cad0-baac-45db-a8e4-63a6e83d00c0-kube-api-access-tfjlm\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2\" (UID: \"4fe9cad0-baac-45db-a8e4-63a6e83d00c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2"
Oct 30 23:00:00.309209 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.309172    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g"]
Oct 30 23:00:00.318559 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.318526    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g"
Oct 30 23:00:00.332671 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.332631    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n"]
Oct 30 23:00:00.336505 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.336472    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:00:00.351943 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.351904    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-rmzqc"]
Oct 30 23:00:00.352119 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.352016    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n"
Oct 30 23:00:00.360023 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.359994    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h"]
Oct 30 23:00:00.360312 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.360286    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-rmzqc"
Oct 30 23:00:00.362886 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.362671    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n"]
Oct 30 23:00:00.367015 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.366985    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn"]
Oct 30 23:00:00.373941 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.373899    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g"]
Oct 30 23:00:00.374143 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.374028    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn"
Oct 30 23:00:00.382587 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.382556    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-rmzqc"]
Oct 30 23:00:00.387250 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.387210    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn"]
Oct 30 23:00:00.391957 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.391928    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx"]
Oct 30 23:00:00.398002 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.397968    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx"]
Oct 30 23:00:00.398152 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.398079    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx"
Oct 30 23:00:00.403110 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.403080    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe9cad0-baac-45db-a8e4-63a6e83d00c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2\" (UID: \"4fe9cad0-baac-45db-a8e4-63a6e83d00c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2"
Oct 30 23:00:00.403299 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.403129    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69be2b84-9d09-48b8-876e-67f49f56d9a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h\" (UID: \"69be2b84-9d09-48b8-876e-67f49f56d9a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h"
Oct 30 23:00:00.403299 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.403160    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q4fb\" (UniqueName: \"kubernetes.io/projected/83c059bc-93eb-4c5e-9803-098ac14602ad-kube-api-access-9q4fb\") pod \"maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g\" (UID: \"83c059bc-93eb-4c5e-9803-098ac14602ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g"
Oct 30 23:00:00.403299 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.403204    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcmx9\" (UniqueName: \"kubernetes.io/projected/69be2b84-9d09-48b8-876e-67f49f56d9a6-kube-api-access-kcmx9\") pod \"maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h\" (UID: \"69be2b84-9d09-48b8-876e-67f49f56d9a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h"
Oct 30 23:00:00.403462 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.403317    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfjlm\" (UniqueName: \"kubernetes.io/projected/4fe9cad0-baac-45db-a8e4-63a6e83d00c0-kube-api-access-tfjlm\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2\" (UID: \"4fe9cad0-baac-45db-a8e4-63a6e83d00c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2"
Oct 30 23:00:00.403462 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.403422    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c059bc-93eb-4c5e-9803-098ac14602ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g\" (UID: \"83c059bc-93eb-4c5e-9803-098ac14602ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g"
Oct 30 23:00:00.403886 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.403863    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe9cad0-baac-45db-a8e4-63a6e83d00c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2\" (UID: \"4fe9cad0-baac-45db-a8e4-63a6e83d00c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2"
Oct 30 23:00:00.425311 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.425269    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfjlm\" (UniqueName: \"kubernetes.io/projected/4fe9cad0-baac-45db-a8e4-63a6e83d00c0-kube-api-access-tfjlm\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2\" (UID: \"4fe9cad0-baac-45db-a8e4-63a6e83d00c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2"
Oct 30 23:00:00.505539 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.505501    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djb6x\" (UniqueName: \"kubernetes.io/projected/c9a31620-90fb-478e-b32a-f34d8ab55384-kube-api-access-djb6x\") pod \"maas-api-key-cleanup-29890020-rmzqc\" (UID: \"c9a31620-90fb-478e-b32a-f34d8ab55384\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-rmzqc"
Oct 30 23:00:00.505771 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.505556    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c059bc-93eb-4c5e-9803-098ac14602ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g\" (UID: \"83c059bc-93eb-4c5e-9803-098ac14602ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g"
Oct 30 23:00:00.505771 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.505625    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56702f85-adba-4f54-a21d-1f4ef1ad4987-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx\" (UID: \"56702f85-adba-4f54-a21d-1f4ef1ad4987\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx"
Oct 30 23:00:00.505771 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.505671    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn5zn\" (UniqueName: \"kubernetes.io/projected/73fcb5fd-f6ef-4313-a95e-993ca08672b9-kube-api-access-mn5zn\") pod \"maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n\" (UID: \"73fcb5fd-f6ef-4313-a95e-993ca08672b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n"
Oct 30 23:00:00.505962 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.505823    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73fcb5fd-f6ef-4313-a95e-993ca08672b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n\" (UID: \"73fcb5fd-f6ef-4313-a95e-993ca08672b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n"
Oct 30 23:00:00.506063 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.506033    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9a31620-90fb-478e-b32a-f34d8ab55384-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890020-rmzqc\" (UID: \"c9a31620-90fb-478e-b32a-f34d8ab55384\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-rmzqc"
Oct 30 23:00:00.506202 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.506090    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b89aa2e2-f770-4bb7-903e-f3c2b4878359-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn\" (UID: \"b89aa2e2-f770-4bb7-903e-f3c2b4878359\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn"
Oct 30 23:00:00.506202 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.506166    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69be2b84-9d09-48b8-876e-67f49f56d9a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h\" (UID: \"69be2b84-9d09-48b8-876e-67f49f56d9a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h"
Oct 30 23:00:00.506397 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.506210    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q4fb\" (UniqueName: \"kubernetes.io/projected/83c059bc-93eb-4c5e-9803-098ac14602ad-kube-api-access-9q4fb\") pod \"maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g\" (UID: \"83c059bc-93eb-4c5e-9803-098ac14602ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g"
Oct 30 23:00:00.506397 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.506258    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v6k6\" (UniqueName: \"kubernetes.io/projected/b89aa2e2-f770-4bb7-903e-f3c2b4878359-kube-api-access-9v6k6\") pod \"maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn\" (UID: \"b89aa2e2-f770-4bb7-903e-f3c2b4878359\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn"
Oct 30 23:00:00.506397 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.506278    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c059bc-93eb-4c5e-9803-098ac14602ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g\" (UID: \"83c059bc-93eb-4c5e-9803-098ac14602ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g"
Oct 30 23:00:00.506397 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.506334    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcmx9\" (UniqueName: \"kubernetes.io/projected/69be2b84-9d09-48b8-876e-67f49f56d9a6-kube-api-access-kcmx9\") pod \"maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h\" (UID: \"69be2b84-9d09-48b8-876e-67f49f56d9a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h"
Oct 30 23:00:00.506653 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.506402    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jg46\" (UniqueName: \"kubernetes.io/projected/56702f85-adba-4f54-a21d-1f4ef1ad4987-kube-api-access-9jg46\") pod \"maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx\" (UID: \"56702f85-adba-4f54-a21d-1f4ef1ad4987\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx"
Oct 30 23:00:00.506892 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.506866    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69be2b84-9d09-48b8-876e-67f49f56d9a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h\" (UID: \"69be2b84-9d09-48b8-876e-67f49f56d9a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h"
Oct 30 23:00:00.529585 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.529545    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcmx9\" (UniqueName: \"kubernetes.io/projected/69be2b84-9d09-48b8-876e-67f49f56d9a6-kube-api-access-kcmx9\") pod \"maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h\" (UID: \"69be2b84-9d09-48b8-876e-67f49f56d9a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h"
Oct 30 23:00:00.530027 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.530007    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q4fb\" (UniqueName: \"kubernetes.io/projected/83c059bc-93eb-4c5e-9803-098ac14602ad-kube-api-access-9q4fb\") pod \"maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g\" (UID: \"83c059bc-93eb-4c5e-9803-098ac14602ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g"
Oct 30 23:00:00.587287 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.587197    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2"
Oct 30 23:00:00.607640 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.607603    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v6k6\" (UniqueName: \"kubernetes.io/projected/b89aa2e2-f770-4bb7-903e-f3c2b4878359-kube-api-access-9v6k6\") pod \"maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn\" (UID: \"b89aa2e2-f770-4bb7-903e-f3c2b4878359\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn"
Oct 30 23:00:00.607804 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.607699    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jg46\" (UniqueName: \"kubernetes.io/projected/56702f85-adba-4f54-a21d-1f4ef1ad4987-kube-api-access-9jg46\") pod \"maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx\" (UID: \"56702f85-adba-4f54-a21d-1f4ef1ad4987\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx"
Oct 30 23:00:00.607804 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.607777    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djb6x\" (UniqueName: \"kubernetes.io/projected/c9a31620-90fb-478e-b32a-f34d8ab55384-kube-api-access-djb6x\") pod \"maas-api-key-cleanup-29890020-rmzqc\" (UID: \"c9a31620-90fb-478e-b32a-f34d8ab55384\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-rmzqc"
Oct 30 23:00:00.607926 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.607819    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56702f85-adba-4f54-a21d-1f4ef1ad4987-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx\" (UID: \"56702f85-adba-4f54-a21d-1f4ef1ad4987\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx"
Oct 30 23:00:00.607926 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.607848    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn5zn\" (UniqueName: \"kubernetes.io/projected/73fcb5fd-f6ef-4313-a95e-993ca08672b9-kube-api-access-mn5zn\") pod \"maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n\" (UID: \"73fcb5fd-f6ef-4313-a95e-993ca08672b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n"
Oct 30 23:00:00.608042 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.607922    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73fcb5fd-f6ef-4313-a95e-993ca08672b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n\" (UID: \"73fcb5fd-f6ef-4313-a95e-993ca08672b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n"
Oct 30 23:00:00.608104 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.608065    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9a31620-90fb-478e-b32a-f34d8ab55384-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890020-rmzqc\" (UID: \"c9a31620-90fb-478e-b32a-f34d8ab55384\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-rmzqc"
Oct 30 23:00:00.608169 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.608121    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b89aa2e2-f770-4bb7-903e-f3c2b4878359-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn\" (UID: \"b89aa2e2-f770-4bb7-903e-f3c2b4878359\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn"
Oct 30 23:00:00.608703 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.608671    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56702f85-adba-4f54-a21d-1f4ef1ad4987-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx\" (UID: \"56702f85-adba-4f54-a21d-1f4ef1ad4987\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx"
Oct 30 23:00:00.608810 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.608743    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73fcb5fd-f6ef-4313-a95e-993ca08672b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n\" (UID: \"73fcb5fd-f6ef-4313-a95e-993ca08672b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n"
Oct 30 23:00:00.608877 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.608859    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b89aa2e2-f770-4bb7-903e-f3c2b4878359-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn\" (UID: \"b89aa2e2-f770-4bb7-903e-f3c2b4878359\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn"
Oct 30 23:00:00.608917 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.608856    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9a31620-90fb-478e-b32a-f34d8ab55384-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890020-rmzqc\" (UID: \"c9a31620-90fb-478e-b32a-f34d8ab55384\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-rmzqc"
Oct 30 23:00:00.633188 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.633150    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jg46\" (UniqueName: \"kubernetes.io/projected/56702f85-adba-4f54-a21d-1f4ef1ad4987-kube-api-access-9jg46\") pod \"maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx\" (UID: \"56702f85-adba-4f54-a21d-1f4ef1ad4987\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx"
Oct 30 23:00:00.633427 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.633369    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v6k6\" (UniqueName: \"kubernetes.io/projected/b89aa2e2-f770-4bb7-903e-f3c2b4878359-kube-api-access-9v6k6\") pod \"maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn\" (UID: \"b89aa2e2-f770-4bb7-903e-f3c2b4878359\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn"
Oct 30 23:00:00.633752 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.633733    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djb6x\" (UniqueName: \"kubernetes.io/projected/c9a31620-90fb-478e-b32a-f34d8ab55384-kube-api-access-djb6x\") pod \"maas-api-key-cleanup-29890020-rmzqc\" (UID: \"c9a31620-90fb-478e-b32a-f34d8ab55384\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-rmzqc"
Oct 30 23:00:00.633896 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.633877    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn5zn\" (UniqueName: \"kubernetes.io/projected/73fcb5fd-f6ef-4313-a95e-993ca08672b9-kube-api-access-mn5zn\") pod \"maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n\" (UID: \"73fcb5fd-f6ef-4313-a95e-993ca08672b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n"
Oct 30 23:00:00.652341 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.650784    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h"
Oct 30 23:00:00.653491 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.653460    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g"
Oct 30 23:00:00.676696 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.676546    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n"
Oct 30 23:00:00.686670 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.686322    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-rmzqc"
Oct 30 23:00:00.695944 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.695048    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn"
Oct 30 23:00:00.714868 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.714735    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx"
Oct 30 23:00:00.935346 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:00.932066    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2"]
Oct 30 23:00:01.243831 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:01.243797    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g"]
Oct 30 23:00:01.247443 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:00:01.247416    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c059bc_93eb_4c5e_9803_098ac14602ad.slice/crio-d6dc60f4fa75e605b8a86429aef8121e75da356e3b525f02a7f76d0975f8fa33 WatchSource:0}: Error finding container d6dc60f4fa75e605b8a86429aef8121e75da356e3b525f02a7f76d0975f8fa33: Status 404 returned error can't find the container with id d6dc60f4fa75e605b8a86429aef8121e75da356e3b525f02a7f76d0975f8fa33
Oct 30 23:00:01.262629 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:01.262577    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h"]
Oct 30 23:00:01.264767 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:00:01.264446    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73fcb5fd_f6ef_4313_a95e_993ca08672b9.slice/crio-55aac8bb9038429bd4c7ccf14a9f66b85fc95bd17a4879296355a557e0a997b9 WatchSource:0}: Error finding container 55aac8bb9038429bd4c7ccf14a9f66b85fc95bd17a4879296355a557e0a997b9: Status 404 returned error can't find the container with id 55aac8bb9038429bd4c7ccf14a9f66b85fc95bd17a4879296355a557e0a997b9
Oct 30 23:00:01.269723 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:01.269648    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n"]
Oct 30 23:00:01.337635 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:01.321752    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:01.361143 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:01.361111    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx"]
Oct 30 23:00:01.369013 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:00:01.368987    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56702f85_adba_4f54_a21d_1f4ef1ad4987.slice/crio-8258f062823aa86ab892683c94087a1228794e39d41ed9ea2ed8fde9a87ff9d7 WatchSource:0}: Error finding container 8258f062823aa86ab892683c94087a1228794e39d41ed9ea2ed8fde9a87ff9d7: Status 404 returned error can't find the container with id 8258f062823aa86ab892683c94087a1228794e39d41ed9ea2ed8fde9a87ff9d7
Oct 30 23:00:01.375027 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:01.374843    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-rmzqc"]
Oct 30 23:00:01.378989 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:01.378963    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn"]
Oct 30 23:00:01.382650 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:00:01.382613    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb89aa2e2_f770_4bb7_903e_f3c2b4878359.slice/crio-57bd76fd3ff2367c7fa55b88363fa174d9aea3e6d240ea3fadeac6cf3e5c7198 WatchSource:0}: Error finding container 57bd76fd3ff2367c7fa55b88363fa174d9aea3e6d240ea3fadeac6cf3e5c7198: Status 404 returned error can't find the container with id 57bd76fd3ff2367c7fa55b88363fa174d9aea3e6d240ea3fadeac6cf3e5c7198
Oct 30 23:00:01.744504 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:01.744466    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx" event={"ID":"56702f85-adba-4f54-a21d-1f4ef1ad4987","Type":"ContainerStarted","Data":"8258f062823aa86ab892683c94087a1228794e39d41ed9ea2ed8fde9a87ff9d7"}
Oct 30 23:00:01.746430 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:01.746394    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-rmzqc" event={"ID":"c9a31620-90fb-478e-b32a-f34d8ab55384","Type":"ContainerStarted","Data":"1cdef217d828ade5496442f93c255be2a302ad5c36906d26bde789cd859c7f04"}
Oct 30 23:00:01.748267 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:01.748217    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n" event={"ID":"73fcb5fd-f6ef-4313-a95e-993ca08672b9","Type":"ContainerStarted","Data":"55aac8bb9038429bd4c7ccf14a9f66b85fc95bd17a4879296355a557e0a997b9"}
Oct 30 23:00:01.750201 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:01.750174    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2" event={"ID":"4fe9cad0-baac-45db-a8e4-63a6e83d00c0","Type":"ContainerStarted","Data":"78546fb7eb1aa9c0ee3f370c24034880125ac927385ae7ee2ea47bbd2af852e0"}
Oct 30 23:00:01.750348 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:01.750240    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2" event={"ID":"4fe9cad0-baac-45db-a8e4-63a6e83d00c0","Type":"ContainerStarted","Data":"185e6138472a6cced7acc71ba6616dbc84de27edce6d390b81da09d8bc5cb5e5"}
Oct 30 23:00:01.751596 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:01.751571    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn" event={"ID":"b89aa2e2-f770-4bb7-903e-f3c2b4878359","Type":"ContainerStarted","Data":"57bd76fd3ff2367c7fa55b88363fa174d9aea3e6d240ea3fadeac6cf3e5c7198"}
Oct 30 23:00:01.754502 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:01.754435    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h" event={"ID":"69be2b84-9d09-48b8-876e-67f49f56d9a6","Type":"ContainerStarted","Data":"20a2f89dd94a042aafd42f9362194715ddd5bfee1574412558dd7cc0bb11195d"}
Oct 30 23:00:01.761240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:01.756476    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g" event={"ID":"83c059bc-93eb-4c5e-9803-098ac14602ad","Type":"ContainerStarted","Data":"d6dc60f4fa75e605b8a86429aef8121e75da356e3b525f02a7f76d0975f8fa33"}
Oct 30 23:00:01.795670 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:01.795611    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2" podStartSLOduration=1.795595957 podStartE2EDuration="1.795595957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:01.793542495 +0000 UTC m=+3187.149908809" watchObservedRunningTime="2026-10-30 23:00:01.795595957 +0000 UTC m=+3187.151962272"
Oct 30 23:00:02.778889 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:02.778852    4889 generic.go:356] "Generic (PLEG): container finished" podID="69be2b84-9d09-48b8-876e-67f49f56d9a6" containerID="13c2de8b1f2deca9448158da7ad0fe1d0c27ce20bea62a75a6ad62da4854b7fc" exitCode=60
Oct 30 23:00:02.779407 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:02.779356    4889 scope.go:122] "RemoveContainer" containerID="13c2de8b1f2deca9448158da7ad0fe1d0c27ce20bea62a75a6ad62da4854b7fc"
Oct 30 23:00:02.779541 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:02.779516    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h" event={"ID":"69be2b84-9d09-48b8-876e-67f49f56d9a6","Type":"ContainerDied","Data":"13c2de8b1f2deca9448158da7ad0fe1d0c27ce20bea62a75a6ad62da4854b7fc"}
Oct 30 23:00:02.780815 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:02.780792    4889 generic.go:356] "Generic (PLEG): container finished" podID="83c059bc-93eb-4c5e-9803-098ac14602ad" containerID="d2e8bf86e4bf94c8755be9fa259bd34b9135ad68cdb352364b8ba4221316815b" exitCode=60
Oct 30 23:00:02.780913 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:02.780866    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g" event={"ID":"83c059bc-93eb-4c5e-9803-098ac14602ad","Type":"ContainerDied","Data":"d2e8bf86e4bf94c8755be9fa259bd34b9135ad68cdb352364b8ba4221316815b"}
Oct 30 23:00:02.781147 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:02.781128    4889 scope.go:122] "RemoveContainer" containerID="d2e8bf86e4bf94c8755be9fa259bd34b9135ad68cdb352364b8ba4221316815b"
Oct 30 23:00:02.783082 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:02.783059    4889 generic.go:356] "Generic (PLEG): container finished" podID="56702f85-adba-4f54-a21d-1f4ef1ad4987" containerID="4f7440aca431b765de5d34be150f8a36b5bfb1727abd607364a63f7217ae4f0c" exitCode=7
Oct 30 23:00:02.783181 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:02.783162    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx" event={"ID":"56702f85-adba-4f54-a21d-1f4ef1ad4987","Type":"ContainerDied","Data":"4f7440aca431b765de5d34be150f8a36b5bfb1727abd607364a63f7217ae4f0c"}
Oct 30 23:00:02.783408 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:02.783393    4889 scope.go:122] "RemoveContainer" containerID="4f7440aca431b765de5d34be150f8a36b5bfb1727abd607364a63f7217ae4f0c"
Oct 30 23:00:02.785069 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:02.785047    4889 generic.go:356] "Generic (PLEG): container finished" podID="c9a31620-90fb-478e-b32a-f34d8ab55384" containerID="f87474c85d7726b39543941ec2bcd6575b341a16e5edb19c0b6ae645c3452a9b" exitCode=60
Oct 30 23:00:02.785153 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:02.785127    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-rmzqc" event={"ID":"c9a31620-90fb-478e-b32a-f34d8ab55384","Type":"ContainerDied","Data":"f87474c85d7726b39543941ec2bcd6575b341a16e5edb19c0b6ae645c3452a9b"}
Oct 30 23:00:02.785359 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:02.785345    4889 scope.go:122] "RemoveContainer" containerID="f87474c85d7726b39543941ec2bcd6575b341a16e5edb19c0b6ae645c3452a9b"
Oct 30 23:00:02.786871 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:02.786850    4889 generic.go:356] "Generic (PLEG): container finished" podID="73fcb5fd-f6ef-4313-a95e-993ca08672b9" containerID="5210f3454ac30259b1b639dc43dab0a05c2c73d95606a4796afdeb031ed4df79" exitCode=60
Oct 30 23:00:02.786952 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:02.786927    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n" event={"ID":"73fcb5fd-f6ef-4313-a95e-993ca08672b9","Type":"ContainerDied","Data":"5210f3454ac30259b1b639dc43dab0a05c2c73d95606a4796afdeb031ed4df79"}
Oct 30 23:00:02.787136 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:02.787120    4889 scope.go:122] "RemoveContainer" containerID="5210f3454ac30259b1b639dc43dab0a05c2c73d95606a4796afdeb031ed4df79"
Oct 30 23:00:02.788602 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:02.788585    4889 generic.go:356] "Generic (PLEG): container finished" podID="4fe9cad0-baac-45db-a8e4-63a6e83d00c0" containerID="78546fb7eb1aa9c0ee3f370c24034880125ac927385ae7ee2ea47bbd2af852e0" exitCode=60
Oct 30 23:00:02.788700 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:02.788662    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2" event={"ID":"4fe9cad0-baac-45db-a8e4-63a6e83d00c0","Type":"ContainerDied","Data":"78546fb7eb1aa9c0ee3f370c24034880125ac927385ae7ee2ea47bbd2af852e0"}
Oct 30 23:00:02.788917 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:02.788902    4889 scope.go:122] "RemoveContainer" containerID="78546fb7eb1aa9c0ee3f370c24034880125ac927385ae7ee2ea47bbd2af852e0"
Oct 30 23:00:02.790513 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:02.790488    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn" event={"ID":"b89aa2e2-f770-4bb7-903e-f3c2b4878359","Type":"ContainerStarted","Data":"98b7a391dcfb625780fa607984a9213d75ca52ad1190b03d3aef0f4bd2cd6671"}
Oct 30 23:00:02.981934 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:02.981879    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn" podStartSLOduration=2.981863581 podStartE2EDuration="2.981863581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:02.976075542 +0000 UTC m=+3188.332441858" watchObservedRunningTime="2026-10-30 23:00:02.981863581 +0000 UTC m=+3188.338229896"
Oct 30 23:00:03.158838 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:03.158724    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:03.847337 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:03.847282    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx" event={"ID":"56702f85-adba-4f54-a21d-1f4ef1ad4987","Type":"ContainerStarted","Data":"1012fc88fe1a441743516a598ae36c26c870d25e7423c966c2e60ec450c00a2f"}
Oct 30 23:00:03.849659 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:03.849624    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-rmzqc" event={"ID":"c9a31620-90fb-478e-b32a-f34d8ab55384","Type":"ContainerStarted","Data":"6cafc64eb708ebaf1f6ef1101dd489424f0130d5989cb14a14c2bb4b2a31ea3a"}
Oct 30 23:00:03.851652 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:03.851628    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n" event={"ID":"73fcb5fd-f6ef-4313-a95e-993ca08672b9","Type":"ContainerStarted","Data":"24f5986f15c7d3be8204db68eb860b2ed10e3efd86aff1badfe3add000b84619"}
Oct 30 23:00:03.853831 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:03.853805    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2" event={"ID":"4fe9cad0-baac-45db-a8e4-63a6e83d00c0","Type":"ContainerStarted","Data":"d0a2b8cfc8dfe02b4544a15812628657329fcbc0a7315a5c79def4f5fec14736"}
Oct 30 23:00:03.856495 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:03.856457    4889 generic.go:356] "Generic (PLEG): container finished" podID="b89aa2e2-f770-4bb7-903e-f3c2b4878359" containerID="98b7a391dcfb625780fa607984a9213d75ca52ad1190b03d3aef0f4bd2cd6671" exitCode=60
Oct 30 23:00:03.856593 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:03.856512    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn" event={"ID":"b89aa2e2-f770-4bb7-903e-f3c2b4878359","Type":"ContainerDied","Data":"98b7a391dcfb625780fa607984a9213d75ca52ad1190b03d3aef0f4bd2cd6671"}
Oct 30 23:00:03.856818 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:03.856793    4889 scope.go:122] "RemoveContainer" containerID="98b7a391dcfb625780fa607984a9213d75ca52ad1190b03d3aef0f4bd2cd6671"
Oct 30 23:00:03.858927 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:03.858900    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h" event={"ID":"69be2b84-9d09-48b8-876e-67f49f56d9a6","Type":"ContainerStarted","Data":"3ebfb88fdb3a7341f6b502bebe3727460a24d0e8091cf0fa90ef546b405788f1"}
Oct 30 23:00:03.861134 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:03.861116    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g" event={"ID":"83c059bc-93eb-4c5e-9803-098ac14602ad","Type":"ContainerStarted","Data":"05c3269b0a380006550a447ae49b3febdb3896f4e0f9f79ba635f047d643f3d0"}
Oct 30 23:00:03.898000 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:03.897946    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx" podStartSLOduration=3.897932569 podStartE2EDuration="3.897932569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:03.897554749 +0000 UTC m=+3189.253921064" watchObservedRunningTime="2026-10-30 23:00:03.897932569 +0000 UTC m=+3189.254298885"
Oct 30 23:00:04.001078 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.001026    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n" podStartSLOduration=4.001011123 podStartE2EDuration="4.001011123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:03.997692102 +0000 UTC m=+3189.354058418" watchObservedRunningTime="2026-10-30 23:00:04.001011123 +0000 UTC m=+3189.357377439"
Oct 30 23:00:04.107336 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.107207    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-rmzqc" podStartSLOduration=4.107190706 podStartE2EDuration="4.107190706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:04.10385054 +0000 UTC m=+3189.460216856" watchObservedRunningTime="2026-10-30 23:00:04.107190706 +0000 UTC m=+3189.463557021"
Oct 30 23:00:04.155967 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.155911    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h" podStartSLOduration=4.155896272 podStartE2EDuration="4.155896272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:04.151453132 +0000 UTC m=+3189.507819447" watchObservedRunningTime="2026-10-30 23:00:04.155896272 +0000 UTC m=+3189.512262587"
Oct 30 23:00:04.197859 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.197799    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g" podStartSLOduration=4.19778372 podStartE2EDuration="4.19778372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:04.193946825 +0000 UTC m=+3189.550313153" watchObservedRunningTime="2026-10-30 23:00:04.19778372 +0000 UTC m=+3189.554150035"
Oct 30 23:00:04.871896 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.871862    4889 generic.go:356] "Generic (PLEG): container finished" podID="4fe9cad0-baac-45db-a8e4-63a6e83d00c0" containerID="d0a2b8cfc8dfe02b4544a15812628657329fcbc0a7315a5c79def4f5fec14736" exitCode=60
Oct 30 23:00:04.872405 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.871953    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2" event={"ID":"4fe9cad0-baac-45db-a8e4-63a6e83d00c0","Type":"ContainerDied","Data":"d0a2b8cfc8dfe02b4544a15812628657329fcbc0a7315a5c79def4f5fec14736"}
Oct 30 23:00:04.872405 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.872014    4889 scope.go:122] "RemoveContainer" containerID="78546fb7eb1aa9c0ee3f370c24034880125ac927385ae7ee2ea47bbd2af852e0"
Oct 30 23:00:04.872405 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.872295    4889 scope.go:122] "RemoveContainer" containerID="d0a2b8cfc8dfe02b4544a15812628657329fcbc0a7315a5c79def4f5fec14736"
Oct 30 23:00:04.872774 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:00:04.872743    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2_odh-ai-gateway-infra(4fe9cad0-baac-45db-a8e4-63a6e83d00c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2" podUID="4fe9cad0-baac-45db-a8e4-63a6e83d00c0"
Oct 30 23:00:04.874547 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.874374    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn" event={"ID":"b89aa2e2-f770-4bb7-903e-f3c2b4878359","Type":"ContainerStarted","Data":"90da7f67ecffdbb1921d1f3a9d235e41d9fec1de5d591d715627536ab0001a7e"}
Oct 30 23:00:04.876498 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.876472    4889 generic.go:356] "Generic (PLEG): container finished" podID="69be2b84-9d09-48b8-876e-67f49f56d9a6" containerID="3ebfb88fdb3a7341f6b502bebe3727460a24d0e8091cf0fa90ef546b405788f1" exitCode=60
Oct 30 23:00:04.876619 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.876552    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h" event={"ID":"69be2b84-9d09-48b8-876e-67f49f56d9a6","Type":"ContainerDied","Data":"3ebfb88fdb3a7341f6b502bebe3727460a24d0e8091cf0fa90ef546b405788f1"}
Oct 30 23:00:04.876800 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.876787    4889 scope.go:122] "RemoveContainer" containerID="3ebfb88fdb3a7341f6b502bebe3727460a24d0e8091cf0fa90ef546b405788f1"
Oct 30 23:00:04.877155 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:00:04.877130    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h_odh-ai-gateway-infra(69be2b84-9d09-48b8-876e-67f49f56d9a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h" podUID="69be2b84-9d09-48b8-876e-67f49f56d9a6"
Oct 30 23:00:04.878582 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.878558    4889 generic.go:356] "Generic (PLEG): container finished" podID="83c059bc-93eb-4c5e-9803-098ac14602ad" containerID="05c3269b0a380006550a447ae49b3febdb3896f4e0f9f79ba635f047d643f3d0" exitCode=60
Oct 30 23:00:04.878724 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.878636    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g" event={"ID":"83c059bc-93eb-4c5e-9803-098ac14602ad","Type":"ContainerDied","Data":"05c3269b0a380006550a447ae49b3febdb3896f4e0f9f79ba635f047d643f3d0"}
Oct 30 23:00:04.878871 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.878852    4889 scope.go:122] "RemoveContainer" containerID="05c3269b0a380006550a447ae49b3febdb3896f4e0f9f79ba635f047d643f3d0"
Oct 30 23:00:04.879184 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:00:04.879159    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g_odh-ai-gateway-infra(83c059bc-93eb-4c5e-9803-098ac14602ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g" podUID="83c059bc-93eb-4c5e-9803-098ac14602ad"
Oct 30 23:00:04.880858 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.880833    4889 generic.go:356] "Generic (PLEG): container finished" podID="56702f85-adba-4f54-a21d-1f4ef1ad4987" containerID="1012fc88fe1a441743516a598ae36c26c870d25e7423c966c2e60ec450c00a2f" exitCode=7
Oct 30 23:00:04.880970 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.880915    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx" event={"ID":"56702f85-adba-4f54-a21d-1f4ef1ad4987","Type":"ContainerDied","Data":"1012fc88fe1a441743516a598ae36c26c870d25e7423c966c2e60ec450c00a2f"}
Oct 30 23:00:04.881157 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.881140    4889 scope.go:122] "RemoveContainer" containerID="1012fc88fe1a441743516a598ae36c26c870d25e7423c966c2e60ec450c00a2f"
Oct 30 23:00:04.881465 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:00:04.881440    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx_odh-ai-gateway-infra(56702f85-adba-4f54-a21d-1f4ef1ad4987)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx" podUID="56702f85-adba-4f54-a21d-1f4ef1ad4987"
Oct 30 23:00:04.883026 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.883000    4889 generic.go:356] "Generic (PLEG): container finished" podID="c9a31620-90fb-478e-b32a-f34d8ab55384" containerID="6cafc64eb708ebaf1f6ef1101dd489424f0130d5989cb14a14c2bb4b2a31ea3a" exitCode=60
Oct 30 23:00:04.883136 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.883097    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-rmzqc" event={"ID":"c9a31620-90fb-478e-b32a-f34d8ab55384","Type":"ContainerDied","Data":"6cafc64eb708ebaf1f6ef1101dd489424f0130d5989cb14a14c2bb4b2a31ea3a"}
Oct 30 23:00:04.883376 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.883361    4889 scope.go:122] "RemoveContainer" containerID="6cafc64eb708ebaf1f6ef1101dd489424f0130d5989cb14a14c2bb4b2a31ea3a"
Oct 30 23:00:04.883674 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:00:04.883649    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890020-rmzqc_odh-ai-gateway-infra(c9a31620-90fb-478e-b32a-f34d8ab55384)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-rmzqc" podUID="c9a31620-90fb-478e-b32a-f34d8ab55384"
Oct 30 23:00:04.885181 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.885157    4889 generic.go:356] "Generic (PLEG): container finished" podID="73fcb5fd-f6ef-4313-a95e-993ca08672b9" containerID="24f5986f15c7d3be8204db68eb860b2ed10e3efd86aff1badfe3add000b84619" exitCode=60
Oct 30 23:00:04.885295 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.885205    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n" event={"ID":"73fcb5fd-f6ef-4313-a95e-993ca08672b9","Type":"ContainerDied","Data":"24f5986f15c7d3be8204db68eb860b2ed10e3efd86aff1badfe3add000b84619"}
Oct 30 23:00:04.885480 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.885464    4889 scope.go:122] "RemoveContainer" containerID="24f5986f15c7d3be8204db68eb860b2ed10e3efd86aff1badfe3add000b84619"
Oct 30 23:00:04.885743 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:00:04.885725    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n_odh-ai-gateway-infra(73fcb5fd-f6ef-4313-a95e-993ca08672b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n" podUID="73fcb5fd-f6ef-4313-a95e-993ca08672b9"
Oct 30 23:00:04.888027 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.888003    4889 scope.go:122] "RemoveContainer" containerID="13c2de8b1f2deca9448158da7ad0fe1d0c27ce20bea62a75a6ad62da4854b7fc"
Oct 30 23:00:04.905171 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.905141    4889 scope.go:122] "RemoveContainer" containerID="d2e8bf86e4bf94c8755be9fa259bd34b9135ad68cdb352364b8ba4221316815b"
Oct 30 23:00:04.949172 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.949144    4889 scope.go:122] "RemoveContainer" containerID="4f7440aca431b765de5d34be150f8a36b5bfb1727abd607364a63f7217ae4f0c"
Oct 30 23:00:04.964765 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.964736    4889 scope.go:122] "RemoveContainer" containerID="f87474c85d7726b39543941ec2bcd6575b341a16e5edb19c0b6ae645c3452a9b"
Oct 30 23:00:04.981467 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:04.981433    4889 scope.go:122] "RemoveContainer" containerID="5210f3454ac30259b1b639dc43dab0a05c2c73d95606a4796afdeb031ed4df79"
Oct 30 23:00:05.091653 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:05.091615    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:05.896880 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:05.896852    4889 scope.go:122] "RemoveContainer" containerID="d0a2b8cfc8dfe02b4544a15812628657329fcbc0a7315a5c79def4f5fec14736"
Oct 30 23:00:05.897395 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:00:05.897329    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2_odh-ai-gateway-infra(4fe9cad0-baac-45db-a8e4-63a6e83d00c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2" podUID="4fe9cad0-baac-45db-a8e4-63a6e83d00c0"
Oct 30 23:00:05.898658 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:05.898628    4889 generic.go:356] "Generic (PLEG): container finished" podID="b89aa2e2-f770-4bb7-903e-f3c2b4878359" containerID="90da7f67ecffdbb1921d1f3a9d235e41d9fec1de5d591d715627536ab0001a7e" exitCode=60
Oct 30 23:00:05.898796 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:05.898707    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn" event={"ID":"b89aa2e2-f770-4bb7-903e-f3c2b4878359","Type":"ContainerDied","Data":"90da7f67ecffdbb1921d1f3a9d235e41d9fec1de5d591d715627536ab0001a7e"}
Oct 30 23:00:05.898796 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:05.898766    4889 scope.go:122] "RemoveContainer" containerID="98b7a391dcfb625780fa607984a9213d75ca52ad1190b03d3aef0f4bd2cd6671"
Oct 30 23:00:05.898999 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:05.898979    4889 scope.go:122] "RemoveContainer" containerID="90da7f67ecffdbb1921d1f3a9d235e41d9fec1de5d591d715627536ab0001a7e"
Oct 30 23:00:05.899377 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:00:05.899350    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn_odh-ai-gateway-infra(b89aa2e2-f770-4bb7-903e-f3c2b4878359)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn" podUID="b89aa2e2-f770-4bb7-903e-f3c2b4878359"
Oct 30 23:00:05.901210 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:05.901186    4889 scope.go:122] "RemoveContainer" containerID="3ebfb88fdb3a7341f6b502bebe3727460a24d0e8091cf0fa90ef546b405788f1"
Oct 30 23:00:05.901595 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:00:05.901568    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h_odh-ai-gateway-infra(69be2b84-9d09-48b8-876e-67f49f56d9a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h" podUID="69be2b84-9d09-48b8-876e-67f49f56d9a6"
Oct 30 23:00:05.902931 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:05.902915    4889 scope.go:122] "RemoveContainer" containerID="05c3269b0a380006550a447ae49b3febdb3896f4e0f9f79ba635f047d643f3d0"
Oct 30 23:00:05.903270 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:00:05.903209    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g_odh-ai-gateway-infra(83c059bc-93eb-4c5e-9803-098ac14602ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g" podUID="83c059bc-93eb-4c5e-9803-098ac14602ad"
Oct 30 23:00:05.905056 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:05.905033    4889 scope.go:122] "RemoveContainer" containerID="1012fc88fe1a441743516a598ae36c26c870d25e7423c966c2e60ec450c00a2f"
Oct 30 23:00:05.905422 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:00:05.905397    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx_odh-ai-gateway-infra(56702f85-adba-4f54-a21d-1f4ef1ad4987)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx" podUID="56702f85-adba-4f54-a21d-1f4ef1ad4987"
Oct 30 23:00:05.906889 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:05.906869    4889 scope.go:122] "RemoveContainer" containerID="6cafc64eb708ebaf1f6ef1101dd489424f0130d5989cb14a14c2bb4b2a31ea3a"
Oct 30 23:00:05.907182 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:00:05.907158    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890020-rmzqc_odh-ai-gateway-infra(c9a31620-90fb-478e-b32a-f34d8ab55384)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-rmzqc" podUID="c9a31620-90fb-478e-b32a-f34d8ab55384"
Oct 30 23:00:05.908718 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:05.908702    4889 scope.go:122] "RemoveContainer" containerID="24f5986f15c7d3be8204db68eb860b2ed10e3efd86aff1badfe3add000b84619"
Oct 30 23:00:05.908930 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:00:05.908913    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n_odh-ai-gateway-infra(73fcb5fd-f6ef-4313-a95e-993ca08672b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n" podUID="73fcb5fd-f6ef-4313-a95e-993ca08672b9"
Oct 30 23:00:06.920495 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:06.920465    4889 scope.go:122] "RemoveContainer" containerID="90da7f67ecffdbb1921d1f3a9d235e41d9fec1de5d591d715627536ab0001a7e"
Oct 30 23:00:06.920903 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:00:06.920856    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn_odh-ai-gateway-infra(b89aa2e2-f770-4bb7-903e-f3c2b4878359)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn" podUID="b89aa2e2-f770-4bb7-903e-f3c2b4878359"
Oct 30 23:00:07.415038 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:07.414979    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:08.044465 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:08.044424    4889 patch_prober.go:29] interesting pod/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:00:08.044992 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:08.044518    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2" podUID="1aa5f50b-7768-46fd-9ce4-76c47268bce2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:00:09.123900 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:09.123861    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:10.306874 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:00:10.306830    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:00:10.764986 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:10.762619    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:11.966145 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:11.966113    4889 generic.go:356] "Generic (PLEG): container finished" podID="1aa5f50b-7768-46fd-9ce4-76c47268bce2" containerID="60c45600e6986249c58dfc88800b63f8a6711b0db3111f5f480ece96a3b8be40" exitCode=0
Oct 30 23:00:11.966778 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:11.966196    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2" event={"ID":"1aa5f50b-7768-46fd-9ce4-76c47268bce2","Type":"ContainerDied","Data":"60c45600e6986249c58dfc88800b63f8a6711b0db3111f5f480ece96a3b8be40"}
Oct 30 23:00:11.966851 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:11.966802    4889 scope.go:122] "RemoveContainer" containerID="60c45600e6986249c58dfc88800b63f8a6711b0db3111f5f480ece96a3b8be40"
Oct 30 23:00:11.968938 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:11.968899    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:00:11.969077 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:11.968981    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:00:11.969077 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:11.969015    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:00:12.523393 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:12.521753    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:12.980557 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:12.980462    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2" event={"ID":"1aa5f50b-7768-46fd-9ce4-76c47268bce2","Type":"ContainerStarted","Data":"df043c0e8586f7d3805171f6196ccf639a8fb53fd6c54891b96fb103e1b34bbd"}
Oct 30 23:00:13.106442 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:13.106408    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 23:00:13.106618 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:13.106546    4889 patch_prober.go:29] interesting pod/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Oct 30 23:00:13.106618 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:13.106591    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2" podUID="1aa5f50b-7768-46fd-9ce4-76c47268bce2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Oct 30 23:00:14.106186 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:14.106148    4889 patch_prober.go:29] interesting pod/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Oct 30 23:00:14.106648 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:14.106233    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2" podUID="1aa5f50b-7768-46fd-9ce4-76c47268bce2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Oct 30 23:00:14.445315 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:14.442817    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:14.773426 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:14.773391    4889 scope.go:122] "RemoveContainer" containerID="3ebfb88fdb3a7341f6b502bebe3727460a24d0e8091cf0fa90ef546b405788f1"
Oct 30 23:00:15.106645 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:15.106545    4889 patch_prober.go:29] interesting pod/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Oct 30 23:00:15.106645 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:15.106608    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2" podUID="1aa5f50b-7768-46fd-9ce4-76c47268bce2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Oct 30 23:00:15.766939 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:15.766903    4889 scope.go:122] "RemoveContainer" containerID="d0a2b8cfc8dfe02b4544a15812628657329fcbc0a7315a5c79def4f5fec14736"
Oct 30 23:00:15.767169 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:15.767080    4889 scope.go:122] "RemoveContainer" containerID="6cafc64eb708ebaf1f6ef1101dd489424f0130d5989cb14a14c2bb4b2a31ea3a"
Oct 30 23:00:16.016180 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:16.016138    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h" event={"ID":"69be2b84-9d09-48b8-876e-67f49f56d9a6","Type":"ContainerStarted","Data":"6e969e28d27b13b89cd625dcd64bbaa5e686497970b447720f62c65dc9ff1c10"}
Oct 30 23:00:16.040678 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:16.040642    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:16.107068 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:16.107032    4889 patch_prober.go:29] interesting pod/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Oct 30 23:00:16.107536 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:16.107091    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2" podUID="1aa5f50b-7768-46fd-9ce4-76c47268bce2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Oct 30 23:00:16.239754 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:16.239577    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:16.441289 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:16.440959    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:16.686899 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:16.686798    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:16.774391 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:16.768744    4889 scope.go:122] "RemoveContainer" containerID="05c3269b0a380006550a447ae49b3febdb3896f4e0f9f79ba635f047d643f3d0"
Oct 30 23:00:16.774391 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:16.769874    4889 scope.go:122] "RemoveContainer" containerID="1012fc88fe1a441743516a598ae36c26c870d25e7423c966c2e60ec450c00a2f"
Oct 30 23:00:16.944052 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:16.943965    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:17.048756 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:17.048721    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-rmzqc" event={"ID":"c9a31620-90fb-478e-b32a-f34d8ab55384","Type":"ContainerStarted","Data":"dadc6d258066afb54b7d48a675f0220dcd2fb70a785d102ccb8c4d7fc51ba8f8"}
Oct 30 23:00:17.051003 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:17.050976    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2" event={"ID":"4fe9cad0-baac-45db-a8e4-63a6e83d00c0","Type":"ContainerStarted","Data":"2fa15e80a8ed6971623854684998202225a0211b138f6ae945b13b96c7f1cb24"}
Oct 30 23:00:17.053887 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:17.053861    4889 generic.go:356] "Generic (PLEG): container finished" podID="69be2b84-9d09-48b8-876e-67f49f56d9a6" containerID="6e969e28d27b13b89cd625dcd64bbaa5e686497970b447720f62c65dc9ff1c10" exitCode=60
Oct 30 23:00:17.053977 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:17.053918    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h" event={"ID":"69be2b84-9d09-48b8-876e-67f49f56d9a6","Type":"ContainerDied","Data":"6e969e28d27b13b89cd625dcd64bbaa5e686497970b447720f62c65dc9ff1c10"}
Oct 30 23:00:17.053977 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:17.053957    4889 scope.go:122] "RemoveContainer" containerID="3ebfb88fdb3a7341f6b502bebe3727460a24d0e8091cf0fa90ef546b405788f1"
Oct 30 23:00:17.054364 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:17.054350    4889 scope.go:122] "RemoveContainer" containerID="6e969e28d27b13b89cd625dcd64bbaa5e686497970b447720f62c65dc9ff1c10"
Oct 30 23:00:17.054734 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:00:17.054714    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h_odh-ai-gateway-infra(69be2b84-9d09-48b8-876e-67f49f56d9a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h" podUID="69be2b84-9d09-48b8-876e-67f49f56d9a6"
Oct 30 23:00:17.081665 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:17.081631    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h"]
Oct 30 23:00:17.106352 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:17.106314    4889 patch_prober.go:29] interesting pod/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Oct 30 23:00:17.106544 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:17.106372    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2" podUID="1aa5f50b-7768-46fd-9ce4-76c47268bce2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Oct 30 23:00:17.387177 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:17.387142    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:17.767512 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:17.767475    4889 scope.go:122] "RemoveContainer" containerID="90da7f67ecffdbb1921d1f3a9d235e41d9fec1de5d591d715627536ab0001a7e"
Oct 30 23:00:18.066938 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:18.066897    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g" event={"ID":"83c059bc-93eb-4c5e-9803-098ac14602ad","Type":"ContainerStarted","Data":"ecfce83c7a6d10fe8cfa527ee7d21e82374b4a583961091fa7729e7176d57930"}
Oct 30 23:00:18.069156 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:18.069127    4889 generic.go:356] "Generic (PLEG): container finished" podID="56702f85-adba-4f54-a21d-1f4ef1ad4987" containerID="7b28cc00affa607c64f7dea65a7a37c3ae39be0ea444b172883c3c9eac076528" exitCode=60
Oct 30 23:00:18.069318 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:18.069200    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx" event={"ID":"56702f85-adba-4f54-a21d-1f4ef1ad4987","Type":"ContainerDied","Data":"7b28cc00affa607c64f7dea65a7a37c3ae39be0ea444b172883c3c9eac076528"}
Oct 30 23:00:18.069318 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:18.069279    4889 scope.go:122] "RemoveContainer" containerID="1012fc88fe1a441743516a598ae36c26c870d25e7423c966c2e60ec450c00a2f"
Oct 30 23:00:18.069707 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:18.069690    4889 scope.go:122] "RemoveContainer" containerID="7b28cc00affa607c64f7dea65a7a37c3ae39be0ea444b172883c3c9eac076528"
Oct 30 23:00:18.070051 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:00:18.070028    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx_odh-ai-gateway-infra(56702f85-adba-4f54-a21d-1f4ef1ad4987)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx" podUID="56702f85-adba-4f54-a21d-1f4ef1ad4987"
Oct 30 23:00:18.073599 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:18.073567    4889 generic.go:356] "Generic (PLEG): container finished" podID="c9a31620-90fb-478e-b32a-f34d8ab55384" containerID="dadc6d258066afb54b7d48a675f0220dcd2fb70a785d102ccb8c4d7fc51ba8f8" exitCode=60
Oct 30 23:00:18.073768 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:18.073711    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-rmzqc" event={"ID":"c9a31620-90fb-478e-b32a-f34d8ab55384","Type":"ContainerDied","Data":"dadc6d258066afb54b7d48a675f0220dcd2fb70a785d102ccb8c4d7fc51ba8f8"}
Oct 30 23:00:18.074276 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:18.074240    4889 scope.go:122] "RemoveContainer" containerID="dadc6d258066afb54b7d48a675f0220dcd2fb70a785d102ccb8c4d7fc51ba8f8"
Oct 30 23:00:18.074714 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:00:18.074680    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890020-rmzqc_odh-ai-gateway-infra(c9a31620-90fb-478e-b32a-f34d8ab55384)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-rmzqc" podUID="c9a31620-90fb-478e-b32a-f34d8ab55384"
Oct 30 23:00:18.076775 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:18.076712    4889 generic.go:356] "Generic (PLEG): container finished" podID="4fe9cad0-baac-45db-a8e4-63a6e83d00c0" containerID="2fa15e80a8ed6971623854684998202225a0211b138f6ae945b13b96c7f1cb24" exitCode=60
Oct 30 23:00:18.078162 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:18.076850    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2" event={"ID":"4fe9cad0-baac-45db-a8e4-63a6e83d00c0","Type":"ContainerDied","Data":"2fa15e80a8ed6971623854684998202225a0211b138f6ae945b13b96c7f1cb24"}
Oct 30 23:00:18.078162 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:18.077465    4889 scope.go:122] "RemoveContainer" containerID="2fa15e80a8ed6971623854684998202225a0211b138f6ae945b13b96c7f1cb24"
Oct 30 23:00:18.078162 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:00:18.077996    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2_odh-ai-gateway-infra(4fe9cad0-baac-45db-a8e4-63a6e83d00c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2" podUID="4fe9cad0-baac-45db-a8e4-63a6e83d00c0"
Oct 30 23:00:18.106334 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:18.106209    4889 patch_prober.go:29] interesting pod/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Oct 30 23:00:18.106334 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:18.106323    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2" podUID="1aa5f50b-7768-46fd-9ce4-76c47268bce2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Oct 30 23:00:18.107834 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:18.107772    4889 scope.go:122] "RemoveContainer" containerID="6cafc64eb708ebaf1f6ef1101dd489424f0130d5989cb14a14c2bb4b2a31ea3a"
Oct 30 23:00:18.147723 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:18.147637    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-rmzqc"]
Oct 30 23:00:18.199680 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:18.199649    4889 scope.go:122] "RemoveContainer" containerID="d0a2b8cfc8dfe02b4544a15812628657329fcbc0a7315a5c79def4f5fec14736"
Oct 30 23:00:18.252248 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:18.250874    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2"]
Oct 30 23:00:18.346806 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:18.346712    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h"
Oct 30 23:00:18.402692 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:18.402122    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69be2b84-9d09-48b8-876e-67f49f56d9a6-kube-api-access-kcmx9\" (UniqueName: \"kubernetes.io/projected/69be2b84-9d09-48b8-876e-67f49f56d9a6-kube-api-access-kcmx9\") pod \"69be2b84-9d09-48b8-876e-67f49f56d9a6\" (UID: \"69be2b84-9d09-48b8-876e-67f49f56d9a6\") "
Oct 30 23:00:18.402692 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:18.402190    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69be2b84-9d09-48b8-876e-67f49f56d9a6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69be2b84-9d09-48b8-876e-67f49f56d9a6-maas-api-service-ca\") pod \"69be2b84-9d09-48b8-876e-67f49f56d9a6\" (UID: \"69be2b84-9d09-48b8-876e-67f49f56d9a6\") "
Oct 30 23:00:18.403381 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:18.402959    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69be2b84-9d09-48b8-876e-67f49f56d9a6-maas-api-service-ca" pod "69be2b84-9d09-48b8-876e-67f49f56d9a6" (UID: "69be2b84-9d09-48b8-876e-67f49f56d9a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:18.404881 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:18.404834    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69be2b84-9d09-48b8-876e-67f49f56d9a6-kube-api-access-kcmx9" pod "69be2b84-9d09-48b8-876e-67f49f56d9a6" (UID: "69be2b84-9d09-48b8-876e-67f49f56d9a6"). InnerVolumeSpecName "kube-api-access-kcmx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:18.505148 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:18.505101    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcmx9\" (UniqueName: \"kubernetes.io/projected/69be2b84-9d09-48b8-876e-67f49f56d9a6-kube-api-access-kcmx9\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:00:18.505148 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:18.505149    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69be2b84-9d09-48b8-876e-67f49f56d9a6-maas-api-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:00:18.767150 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:18.767122    4889 scope.go:122] "RemoveContainer" containerID="24f5986f15c7d3be8204db68eb860b2ed10e3efd86aff1badfe3add000b84619"
Oct 30 23:00:19.091468 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.091434    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h"
Oct 30 23:00:19.091468 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.091455    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h" event={"ID":"69be2b84-9d09-48b8-876e-67f49f56d9a6","Type":"ContainerDied","Data":"20a2f89dd94a042aafd42f9362194715ddd5bfee1574412558dd7cc0bb11195d"}
Oct 30 23:00:19.091790 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.091570    4889 scope.go:122] "RemoveContainer" containerID="6e969e28d27b13b89cd625dcd64bbaa5e686497970b447720f62c65dc9ff1c10"
Oct 30 23:00:19.096740 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.096709    4889 generic.go:356] "Generic (PLEG): container finished" podID="83c059bc-93eb-4c5e-9803-098ac14602ad" containerID="ecfce83c7a6d10fe8cfa527ee7d21e82374b4a583961091fa7729e7176d57930" exitCode=60
Oct 30 23:00:19.096928 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.096788    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g" event={"ID":"83c059bc-93eb-4c5e-9803-098ac14602ad","Type":"ContainerDied","Data":"ecfce83c7a6d10fe8cfa527ee7d21e82374b4a583961091fa7729e7176d57930"}
Oct 30 23:00:19.097400 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.097377    4889 scope.go:122] "RemoveContainer" containerID="ecfce83c7a6d10fe8cfa527ee7d21e82374b4a583961091fa7729e7176d57930"
Oct 30 23:00:19.097860 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:00:19.097833    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g_odh-ai-gateway-infra(83c059bc-93eb-4c5e-9803-098ac14602ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g" podUID="83c059bc-93eb-4c5e-9803-098ac14602ad"
Oct 30 23:00:19.101815 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.101780    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:19.111640 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.111590    4889 scope.go:122] "RemoveContainer" containerID="05c3269b0a380006550a447ae49b3febdb3896f4e0f9f79ba635f047d643f3d0"
Oct 30 23:00:19.112684 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.112650    4889 generic.go:356] "Generic (PLEG): container finished" podID="b89aa2e2-f770-4bb7-903e-f3c2b4878359" containerID="8be52f09a6fbc1219feb54273f0bc649a985432fd25e05f1d82f76108b27d5d8" exitCode=60
Oct 30 23:00:19.112838 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.112723    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn" event={"ID":"b89aa2e2-f770-4bb7-903e-f3c2b4878359","Type":"ContainerDied","Data":"8be52f09a6fbc1219feb54273f0bc649a985432fd25e05f1d82f76108b27d5d8"}
Oct 30 23:00:19.114387 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.113414    4889 scope.go:122] "RemoveContainer" containerID="8be52f09a6fbc1219feb54273f0bc649a985432fd25e05f1d82f76108b27d5d8"
Oct 30 23:00:19.114387 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:00:19.113897    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn_odh-ai-gateway-infra(b89aa2e2-f770-4bb7-903e-f3c2b4878359)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn" podUID="b89aa2e2-f770-4bb7-903e-f3c2b4878359"
Oct 30 23:00:19.114387 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.114340    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 23:00:19.114774 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.114719    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 23:00:19.116074 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.116038    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 23:00:19.143839 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.143807    4889 scope.go:122] "RemoveContainer" containerID="90da7f67ecffdbb1921d1f3a9d235e41d9fec1de5d591d715627536ab0001a7e"
Oct 30 23:00:19.205283 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.203931    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h"]
Oct 30 23:00:19.220521 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.220486    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29890020-zjx6h"]
Oct 30 23:00:19.226342 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.226305    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx"]
Oct 30 23:00:19.300796 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.300484    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g"]
Oct 30 23:00:19.383959 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.383929    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2"
Oct 30 23:00:19.390927 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.390909    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-rmzqc"
Oct 30 23:00:19.428298 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.427832    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe9cad0-baac-45db-a8e4-63a6e83d00c0-maas-api-service-ca" pod "4fe9cad0-baac-45db-a8e4-63a6e83d00c0" (UID: "4fe9cad0-baac-45db-a8e4-63a6e83d00c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:19.428823 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.428793    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fe9cad0-baac-45db-a8e4-63a6e83d00c0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe9cad0-baac-45db-a8e4-63a6e83d00c0-maas-api-service-ca\") pod \"4fe9cad0-baac-45db-a8e4-63a6e83d00c0\" (UID: \"4fe9cad0-baac-45db-a8e4-63a6e83d00c0\") "
Oct 30 23:00:19.430694 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.428904    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9a31620-90fb-478e-b32a-f34d8ab55384-kube-api-access-djb6x\" (UniqueName: \"kubernetes.io/projected/c9a31620-90fb-478e-b32a-f34d8ab55384-kube-api-access-djb6x\") pod \"c9a31620-90fb-478e-b32a-f34d8ab55384\" (UID: \"c9a31620-90fb-478e-b32a-f34d8ab55384\") "
Oct 30 23:00:19.430694 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.429053    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe9cad0-baac-45db-a8e4-63a6e83d00c0-kube-api-access-tfjlm\" (UniqueName: \"kubernetes.io/projected/4fe9cad0-baac-45db-a8e4-63a6e83d00c0-kube-api-access-tfjlm\") pod \"4fe9cad0-baac-45db-a8e4-63a6e83d00c0\" (UID: \"4fe9cad0-baac-45db-a8e4-63a6e83d00c0\") "
Oct 30 23:00:19.430694 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.429547    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9a31620-90fb-478e-b32a-f34d8ab55384-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9a31620-90fb-478e-b32a-f34d8ab55384-maas-api-service-ca\") pod \"c9a31620-90fb-478e-b32a-f34d8ab55384\" (UID: \"c9a31620-90fb-478e-b32a-f34d8ab55384\") "
Oct 30 23:00:19.430986 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.430778    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9a31620-90fb-478e-b32a-f34d8ab55384-maas-api-service-ca" pod "c9a31620-90fb-478e-b32a-f34d8ab55384" (UID: "c9a31620-90fb-478e-b32a-f34d8ab55384"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:19.433754 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.433725    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe9cad0-baac-45db-a8e4-63a6e83d00c0-maas-api-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:00:19.433906 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.433759    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9a31620-90fb-478e-b32a-f34d8ab55384-maas-api-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:00:19.450274 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.448315    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe9cad0-baac-45db-a8e4-63a6e83d00c0-kube-api-access-tfjlm" pod "4fe9cad0-baac-45db-a8e4-63a6e83d00c0" (UID: "4fe9cad0-baac-45db-a8e4-63a6e83d00c0"). InnerVolumeSpecName "kube-api-access-tfjlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:19.450274 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.449822    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a31620-90fb-478e-b32a-f34d8ab55384-kube-api-access-djb6x" pod "c9a31620-90fb-478e-b32a-f34d8ab55384" (UID: "c9a31620-90fb-478e-b32a-f34d8ab55384"). InnerVolumeSpecName "kube-api-access-djb6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:19.497606 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.497573    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx"
Oct 30 23:00:19.534675 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.534575    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56702f85-adba-4f54-a21d-1f4ef1ad4987-kube-api-access-9jg46\" (UniqueName: \"kubernetes.io/projected/56702f85-adba-4f54-a21d-1f4ef1ad4987-kube-api-access-9jg46\") pod \"56702f85-adba-4f54-a21d-1f4ef1ad4987\" (UID: \"56702f85-adba-4f54-a21d-1f4ef1ad4987\") "
Oct 30 23:00:19.534895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.534686    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56702f85-adba-4f54-a21d-1f4ef1ad4987-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56702f85-adba-4f54-a21d-1f4ef1ad4987-maas-api-service-ca\") pod \"56702f85-adba-4f54-a21d-1f4ef1ad4987\" (UID: \"56702f85-adba-4f54-a21d-1f4ef1ad4987\") "
Oct 30 23:00:19.535543 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.535372    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfjlm\" (UniqueName: \"kubernetes.io/projected/4fe9cad0-baac-45db-a8e4-63a6e83d00c0-kube-api-access-tfjlm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:00:19.535543 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.535395    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djb6x\" (UniqueName: \"kubernetes.io/projected/c9a31620-90fb-478e-b32a-f34d8ab55384-kube-api-access-djb6x\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:00:19.535543 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.535495    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56702f85-adba-4f54-a21d-1f4ef1ad4987-maas-api-service-ca" pod "56702f85-adba-4f54-a21d-1f4ef1ad4987" (UID: "56702f85-adba-4f54-a21d-1f4ef1ad4987"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:19.538898 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.538860    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56702f85-adba-4f54-a21d-1f4ef1ad4987-kube-api-access-9jg46" pod "56702f85-adba-4f54-a21d-1f4ef1ad4987" (UID: "56702f85-adba-4f54-a21d-1f4ef1ad4987"). InnerVolumeSpecName "kube-api-access-9jg46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:19.636701 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.636590    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jg46\" (UniqueName: \"kubernetes.io/projected/56702f85-adba-4f54-a21d-1f4ef1ad4987-kube-api-access-9jg46\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:00:19.636902 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:19.636844    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56702f85-adba-4f54-a21d-1f4ef1ad4987-maas-api-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:00:20.125878 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.125849    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-rmzqc"
Oct 30 23:00:20.126083 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.125846    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-rmzqc" event={"ID":"c9a31620-90fb-478e-b32a-f34d8ab55384","Type":"ContainerDied","Data":"1cdef217d828ade5496442f93c255be2a302ad5c36906d26bde789cd859c7f04"}
Oct 30 23:00:20.126083 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.125983    4889 scope.go:122] "RemoveContainer" containerID="dadc6d258066afb54b7d48a675f0220dcd2fb70a785d102ccb8c4d7fc51ba8f8"
Oct 30 23:00:20.128342 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.128311    4889 generic.go:356] "Generic (PLEG): container finished" podID="73fcb5fd-f6ef-4313-a95e-993ca08672b9" containerID="92c5a26c1cb1230ad80f25066bc7f7489323a58790b69dd29e66121138449f40" exitCode=60
Oct 30 23:00:20.128460 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.128387    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n" event={"ID":"73fcb5fd-f6ef-4313-a95e-993ca08672b9","Type":"ContainerDied","Data":"92c5a26c1cb1230ad80f25066bc7f7489323a58790b69dd29e66121138449f40"}
Oct 30 23:00:20.128985 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.128964    4889 scope.go:122] "RemoveContainer" containerID="92c5a26c1cb1230ad80f25066bc7f7489323a58790b69dd29e66121138449f40"
Oct 30 23:00:20.129500 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:00:20.129469    4889 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n_odh-ai-gateway-infra(73fcb5fd-f6ef-4313-a95e-993ca08672b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n" podUID="73fcb5fd-f6ef-4313-a95e-993ca08672b9"
Oct 30 23:00:20.130484 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.130246    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2"
Oct 30 23:00:20.130484 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.130276    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2" event={"ID":"4fe9cad0-baac-45db-a8e4-63a6e83d00c0","Type":"ContainerDied","Data":"185e6138472a6cced7acc71ba6616dbc84de27edce6d390b81da09d8bc5cb5e5"}
Oct 30 23:00:20.141770 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.141706    4889 scope.go:122] "RemoveContainer" containerID="24f5986f15c7d3be8204db68eb860b2ed10e3efd86aff1badfe3add000b84619"
Oct 30 23:00:20.145023 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.143577    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx" event={"ID":"56702f85-adba-4f54-a21d-1f4ef1ad4987","Type":"ContainerDied","Data":"8258f062823aa86ab892683c94087a1228794e39d41ed9ea2ed8fde9a87ff9d7"}
Oct 30 23:00:20.145023 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.143978    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx"
Oct 30 23:00:20.166019 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.165978    4889 scope.go:122] "RemoveContainer" containerID="2fa15e80a8ed6971623854684998202225a0211b138f6ae945b13b96c7f1cb24"
Oct 30 23:00:20.183351 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.183185    4889 scope.go:122] "RemoveContainer" containerID="7b28cc00affa607c64f7dea65a7a37c3ae39be0ea444b172883c3c9eac076528"
Oct 30 23:00:20.260304 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.260267    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-rmzqc"]
Oct 30 23:00:20.264625 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.264590    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-rmzqc"]
Oct 30 23:00:20.340000 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.339970    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2"]
Oct 30 23:00:20.345532 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.345498    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890020-gqbw2"]
Oct 30 23:00:20.350237 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.350186    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g"
Oct 30 23:00:20.380807 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.380774    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx"]
Oct 30 23:00:20.385243 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.385193    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdfa86-29890020-w6bcx"]
Oct 30 23:00:20.452314 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.452194    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83c059bc-93eb-4c5e-9803-098ac14602ad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c059bc-93eb-4c5e-9803-098ac14602ad-maas-api-service-ca\") pod \"83c059bc-93eb-4c5e-9803-098ac14602ad\" (UID: \"83c059bc-93eb-4c5e-9803-098ac14602ad\") "
Oct 30 23:00:20.452724 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.452313    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c059bc-93eb-4c5e-9803-098ac14602ad-kube-api-access-9q4fb\" (UniqueName: \"kubernetes.io/projected/83c059bc-93eb-4c5e-9803-098ac14602ad-kube-api-access-9q4fb\") pod \"83c059bc-93eb-4c5e-9803-098ac14602ad\" (UID: \"83c059bc-93eb-4c5e-9803-098ac14602ad\") "
Oct 30 23:00:20.453239 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.453191    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c059bc-93eb-4c5e-9803-098ac14602ad-maas-api-service-ca" pod "83c059bc-93eb-4c5e-9803-098ac14602ad" (UID: "83c059bc-93eb-4c5e-9803-098ac14602ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:20.453662 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.453640    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn"]
Oct 30 23:00:20.455403 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.455370    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c059bc-93eb-4c5e-9803-098ac14602ad-kube-api-access-9q4fb" pod "83c059bc-93eb-4c5e-9803-098ac14602ad" (UID: "83c059bc-93eb-4c5e-9803-098ac14602ad"). InnerVolumeSpecName "kube-api-access-9q4fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:20.556270 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.554913    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c059bc-93eb-4c5e-9803-098ac14602ad-maas-api-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:00:20.556270 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.554957    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q4fb\" (UniqueName: \"kubernetes.io/projected/83c059bc-93eb-4c5e-9803-098ac14602ad-kube-api-access-9q4fb\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:00:20.625179 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.625145    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn"
Oct 30 23:00:20.655614 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.655575    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b89aa2e2-f770-4bb7-903e-f3c2b4878359-kube-api-access-9v6k6\" (UniqueName: \"kubernetes.io/projected/b89aa2e2-f770-4bb7-903e-f3c2b4878359-kube-api-access-9v6k6\") pod \"b89aa2e2-f770-4bb7-903e-f3c2b4878359\" (UID: \"b89aa2e2-f770-4bb7-903e-f3c2b4878359\") "
Oct 30 23:00:20.655800 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.655674    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b89aa2e2-f770-4bb7-903e-f3c2b4878359-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b89aa2e2-f770-4bb7-903e-f3c2b4878359-maas-api-service-ca\") pod \"b89aa2e2-f770-4bb7-903e-f3c2b4878359\" (UID: \"b89aa2e2-f770-4bb7-903e-f3c2b4878359\") "
Oct 30 23:00:20.656060 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.656035    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b89aa2e2-f770-4bb7-903e-f3c2b4878359-maas-api-service-ca" pod "b89aa2e2-f770-4bb7-903e-f3c2b4878359" (UID: "b89aa2e2-f770-4bb7-903e-f3c2b4878359"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:20.656382 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.656366    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b89aa2e2-f770-4bb7-903e-f3c2b4878359-maas-api-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:00:20.658143 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.658112    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b89aa2e2-f770-4bb7-903e-f3c2b4878359-kube-api-access-9v6k6" pod "b89aa2e2-f770-4bb7-903e-f3c2b4878359" (UID: "b89aa2e2-f770-4bb7-903e-f3c2b4878359"). InnerVolumeSpecName "kube-api-access-9v6k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:20.757603 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.757559    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v6k6\" (UniqueName: \"kubernetes.io/projected/b89aa2e2-f770-4bb7-903e-f3c2b4878359-kube-api-access-9v6k6\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:00:20.778152 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.778113    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fe9cad0-baac-45db-a8e4-63a6e83d00c0" path="/var/lib/kubelet/pods/4fe9cad0-baac-45db-a8e4-63a6e83d00c0/volumes"
Oct 30 23:00:20.778738 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.778719    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56702f85-adba-4f54-a21d-1f4ef1ad4987" path="/var/lib/kubelet/pods/56702f85-adba-4f54-a21d-1f4ef1ad4987/volumes"
Oct 30 23:00:20.779132 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.779119    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69be2b84-9d09-48b8-876e-67f49f56d9a6" path="/var/lib/kubelet/pods/69be2b84-9d09-48b8-876e-67f49f56d9a6/volumes"
Oct 30 23:00:20.779921 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:20.779904    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9a31620-90fb-478e-b32a-f34d8ab55384" path="/var/lib/kubelet/pods/c9a31620-90fb-478e-b32a-f34d8ab55384/volumes"
Oct 30 23:00:21.157640 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:21.157605    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g"
Oct 30 23:00:21.157640 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:21.157625    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g" event={"ID":"83c059bc-93eb-4c5e-9803-098ac14602ad","Type":"ContainerDied","Data":"d6dc60f4fa75e605b8a86429aef8121e75da356e3b525f02a7f76d0975f8fa33"}
Oct 30 23:00:21.158021 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:21.157716    4889 scope.go:122] "RemoveContainer" containerID="ecfce83c7a6d10fe8cfa527ee7d21e82374b4a583961091fa7729e7176d57930"
Oct 30 23:00:21.167766 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:21.167740    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn"
Oct 30 23:00:21.167968 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:21.167763    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn" event={"ID":"b89aa2e2-f770-4bb7-903e-f3c2b4878359","Type":"ContainerDied","Data":"57bd76fd3ff2367c7fa55b88363fa174d9aea3e6d240ea3fadeac6cf3e5c7198"}
Oct 30 23:00:21.173564 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:21.173504    4889 scope.go:122] "RemoveContainer" containerID="8be52f09a6fbc1219feb54273f0bc649a985432fd25e05f1d82f76108b27d5d8"
Oct 30 23:00:21.219892 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:21.219856    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn"]
Oct 30 23:00:21.226890 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:21.226843    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n"]
Oct 30 23:00:21.233672 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:21.233599    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f94561-29890020-5mhhn"]
Oct 30 23:00:21.271433 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:21.271388    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g"]
Oct 30 23:00:21.278802 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:21.278741    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29890020-f7b2g"]
Oct 30 23:00:21.438421 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:21.438388    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n"
Oct 30 23:00:21.569763 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:21.569668    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73fcb5fd-f6ef-4313-a95e-993ca08672b9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73fcb5fd-f6ef-4313-a95e-993ca08672b9-maas-api-service-ca\") pod \"73fcb5fd-f6ef-4313-a95e-993ca08672b9\" (UID: \"73fcb5fd-f6ef-4313-a95e-993ca08672b9\") "
Oct 30 23:00:21.570150 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:21.569917    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73fcb5fd-f6ef-4313-a95e-993ca08672b9-kube-api-access-mn5zn\" (UniqueName: \"kubernetes.io/projected/73fcb5fd-f6ef-4313-a95e-993ca08672b9-kube-api-access-mn5zn\") pod \"73fcb5fd-f6ef-4313-a95e-993ca08672b9\" (UID: \"73fcb5fd-f6ef-4313-a95e-993ca08672b9\") "
Oct 30 23:00:21.570150 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:21.570064    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73fcb5fd-f6ef-4313-a95e-993ca08672b9-maas-api-service-ca" pod "73fcb5fd-f6ef-4313-a95e-993ca08672b9" (UID: "73fcb5fd-f6ef-4313-a95e-993ca08672b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:21.570483 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:21.570466    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73fcb5fd-f6ef-4313-a95e-993ca08672b9-maas-api-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:00:21.572303 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:21.572273    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73fcb5fd-f6ef-4313-a95e-993ca08672b9-kube-api-access-mn5zn" pod "73fcb5fd-f6ef-4313-a95e-993ca08672b9" (UID: "73fcb5fd-f6ef-4313-a95e-993ca08672b9"). InnerVolumeSpecName "kube-api-access-mn5zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:21.642726 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:21.642682    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:21.671800 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:21.671762    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn5zn\" (UniqueName: \"kubernetes.io/projected/73fcb5fd-f6ef-4313-a95e-993ca08672b9-kube-api-access-mn5zn\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:00:22.181125 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:22.181092    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n"
Oct 30 23:00:22.181426 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:22.181127    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n" event={"ID":"73fcb5fd-f6ef-4313-a95e-993ca08672b9","Type":"ContainerDied","Data":"55aac8bb9038429bd4c7ccf14a9f66b85fc95bd17a4879296355a557e0a997b9"}
Oct 30 23:00:22.181426 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:22.181189    4889 scope.go:122] "RemoveContainer" containerID="92c5a26c1cb1230ad80f25066bc7f7489323a58790b69dd29e66121138449f40"
Oct 30 23:00:22.242756 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:22.242715    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n"]
Oct 30 23:00:22.256431 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:22.256398    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890020-5s56n"]
Oct 30 23:00:22.399704 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:22.399653    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:22.777744 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:22.777703    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73fcb5fd-f6ef-4313-a95e-993ca08672b9" path="/var/lib/kubelet/pods/73fcb5fd-f6ef-4313-a95e-993ca08672b9/volumes"
Oct 30 23:00:22.778309 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:22.778291    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83c059bc-93eb-4c5e-9803-098ac14602ad" path="/var/lib/kubelet/pods/83c059bc-93eb-4c5e-9803-098ac14602ad/volumes"
Oct 30 23:00:22.778788 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:22.778773    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b89aa2e2-f770-4bb7-903e-f3c2b4878359" path="/var/lib/kubelet/pods/b89aa2e2-f770-4bb7-903e-f3c2b4878359/volumes"
Oct 30 23:00:23.949946 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:23.949901    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:25.709969 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:25.709931    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:26.138790 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:26.138752    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:27.666315 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:27.666217    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:29.464739 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:29.464701    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:29.700758 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:29.700719    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9"]
Oct 30 23:00:29.735539 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:29.735446    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c"]
Oct 30 23:00:29.893121 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:29.893082    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:30.066289 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:30.066252    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:30.235279 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:30.234531    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:31.252357 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:31.252313    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:32.973159 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:32.973118    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:34.652025 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:34.651988    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt"]
Oct 30 23:00:34.701139 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:34.701060    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9" podUID="cb8af940-2412-4e39-a599-450debc2e114" containerName="payload-processing" containerID="cri-o://a5d31ac981a7a9c5c3603cf1c6bfbb0ad9cdce1a09b27c9f64270c9688916d50" gracePeriod=25
Oct 30 23:00:34.743358 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:34.736708    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c" podUID="b1ea4fda-c921-46a1-8107-441bea92acf1" containerName="payload-pre-processing" containerID="cri-o://0ea9d4bef8cba1bd9de15e7344e5aaca13c4be5244f3f7c4b79ec50c5f447a62" gracePeriod=25
Oct 30 23:00:34.832112 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:34.832074    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:35.015494 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.015297    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:35.080760 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.080723    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9"
Oct 30 23:00:35.102383 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.102348    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c"
Oct 30 23:00:35.186916 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.186815    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1ea4fda-c921-46a1-8107-441bea92acf1-kube-api-access-fgml9\" (UniqueName: \"kubernetes.io/projected/b1ea4fda-c921-46a1-8107-441bea92acf1-kube-api-access-fgml9\") pod \"b1ea4fda-c921-46a1-8107-441bea92acf1\" (UID: \"b1ea4fda-c921-46a1-8107-441bea92acf1\") "
Oct 30 23:00:35.187106 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.187049    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1ea4fda-c921-46a1-8107-441bea92acf1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ea4fda-c921-46a1-8107-441bea92acf1-plugins-config-volume\") pod \"b1ea4fda-c921-46a1-8107-441bea92acf1\" (UID: \"b1ea4fda-c921-46a1-8107-441bea92acf1\") "
Oct 30 23:00:35.187214 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.187200    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb8af940-2412-4e39-a599-450debc2e114-kube-api-access-5qrjq\" (UniqueName: \"kubernetes.io/projected/cb8af940-2412-4e39-a599-450debc2e114-kube-api-access-5qrjq\") pod \"cb8af940-2412-4e39-a599-450debc2e114\" (UID: \"cb8af940-2412-4e39-a599-450debc2e114\") "
Oct 30 23:00:35.187303 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.187263    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb8af940-2412-4e39-a599-450debc2e114-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8af940-2412-4e39-a599-450debc2e114-plugins-config-volume\") pod \"cb8af940-2412-4e39-a599-450debc2e114\" (UID: \"cb8af940-2412-4e39-a599-450debc2e114\") "
Oct 30 23:00:35.187450 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.187426    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ea4fda-c921-46a1-8107-441bea92acf1-plugins-config-volume" pod "b1ea4fda-c921-46a1-8107-441bea92acf1" (UID: "b1ea4fda-c921-46a1-8107-441bea92acf1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:35.187770 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.187743    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb8af940-2412-4e39-a599-450debc2e114-plugins-config-volume" pod "cb8af940-2412-4e39-a599-450debc2e114" (UID: "cb8af940-2412-4e39-a599-450debc2e114"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:35.187939 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.187920    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ea4fda-c921-46a1-8107-441bea92acf1-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:00:35.187977 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.187947    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8af940-2412-4e39-a599-450debc2e114-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:00:35.189647 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.189615    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ea4fda-c921-46a1-8107-441bea92acf1-kube-api-access-fgml9" pod "b1ea4fda-c921-46a1-8107-441bea92acf1" (UID: "b1ea4fda-c921-46a1-8107-441bea92acf1"). InnerVolumeSpecName "kube-api-access-fgml9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:35.189766 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.189667    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb8af940-2412-4e39-a599-450debc2e114-kube-api-access-5qrjq" pod "cb8af940-2412-4e39-a599-450debc2e114" (UID: "cb8af940-2412-4e39-a599-450debc2e114"). InnerVolumeSpecName "kube-api-access-5qrjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:35.295246 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.294632    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qrjq\" (UniqueName: \"kubernetes.io/projected/cb8af940-2412-4e39-a599-450debc2e114-kube-api-access-5qrjq\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:00:35.295246 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.294672    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgml9\" (UniqueName: \"kubernetes.io/projected/b1ea4fda-c921-46a1-8107-441bea92acf1-kube-api-access-fgml9\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:00:35.310251 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.309731    4889 generic.go:356] "Generic (PLEG): container finished" podID="b1ea4fda-c921-46a1-8107-441bea92acf1" containerID="0ea9d4bef8cba1bd9de15e7344e5aaca13c4be5244f3f7c4b79ec50c5f447a62" exitCode=0
Oct 30 23:00:35.310251 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.310130    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c" event={"ID":"b1ea4fda-c921-46a1-8107-441bea92acf1","Type":"ContainerDied","Data":"0ea9d4bef8cba1bd9de15e7344e5aaca13c4be5244f3f7c4b79ec50c5f447a62"}
Oct 30 23:00:35.310251 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.310187    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c" event={"ID":"b1ea4fda-c921-46a1-8107-441bea92acf1","Type":"ContainerDied","Data":"668ac023fb4f4adcbcd93f437143aed61c899002bcba20163dcd8c8af247edcf"}
Oct 30 23:00:35.310251 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.310217    4889 scope.go:122] "RemoveContainer" containerID="0ea9d4bef8cba1bd9de15e7344e5aaca13c4be5244f3f7c4b79ec50c5f447a62"
Oct 30 23:00:35.310649 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.310402    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c"
Oct 30 23:00:35.343472 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.339127    4889 generic.go:356] "Generic (PLEG): container finished" podID="cb8af940-2412-4e39-a599-450debc2e114" containerID="a5d31ac981a7a9c5c3603cf1c6bfbb0ad9cdce1a09b27c9f64270c9688916d50" exitCode=0
Oct 30 23:00:35.343472 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.339609    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9"
Oct 30 23:00:35.347249 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.344269    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9" event={"ID":"cb8af940-2412-4e39-a599-450debc2e114","Type":"ContainerDied","Data":"a5d31ac981a7a9c5c3603cf1c6bfbb0ad9cdce1a09b27c9f64270c9688916d50"}
Oct 30 23:00:35.347249 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.344427    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9" event={"ID":"cb8af940-2412-4e39-a599-450debc2e114","Type":"ContainerDied","Data":"3f507f15175e1deb95b1c8434218a4c59582528b2b011b1270bd56f09a143b46"}
Oct 30 23:00:35.395189 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.395159    4889 scope.go:122] "RemoveContainer" containerID="0ea9d4bef8cba1bd9de15e7344e5aaca13c4be5244f3f7c4b79ec50c5f447a62"
Oct 30 23:00:35.395599 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:00:35.395575    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ea9d4bef8cba1bd9de15e7344e5aaca13c4be5244f3f7c4b79ec50c5f447a62\": container with ID starting with 0ea9d4bef8cba1bd9de15e7344e5aaca13c4be5244f3f7c4b79ec50c5f447a62 not found: ID does not exist" containerID="0ea9d4bef8cba1bd9de15e7344e5aaca13c4be5244f3f7c4b79ec50c5f447a62"
Oct 30 23:00:35.395700 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.395607    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea9d4bef8cba1bd9de15e7344e5aaca13c4be5244f3f7c4b79ec50c5f447a62"} err="failed to get container status \"0ea9d4bef8cba1bd9de15e7344e5aaca13c4be5244f3f7c4b79ec50c5f447a62\": rpc error: code = NotFound desc = could not find container \"0ea9d4bef8cba1bd9de15e7344e5aaca13c4be5244f3f7c4b79ec50c5f447a62\": container with ID starting with 0ea9d4bef8cba1bd9de15e7344e5aaca13c4be5244f3f7c4b79ec50c5f447a62 not found: ID does not exist"
Oct 30 23:00:35.395700 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.395628    4889 scope.go:122] "RemoveContainer" containerID="a5d31ac981a7a9c5c3603cf1c6bfbb0ad9cdce1a09b27c9f64270c9688916d50"
Oct 30 23:00:35.405972 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.405936    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c"]
Oct 30 23:00:35.407655 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.407632    4889 scope.go:122] "RemoveContainer" containerID="a5d31ac981a7a9c5c3603cf1c6bfbb0ad9cdce1a09b27c9f64270c9688916d50"
Oct 30 23:00:35.408023 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:00:35.408001    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5d31ac981a7a9c5c3603cf1c6bfbb0ad9cdce1a09b27c9f64270c9688916d50\": container with ID starting with a5d31ac981a7a9c5c3603cf1c6bfbb0ad9cdce1a09b27c9f64270c9688916d50 not found: ID does not exist" containerID="a5d31ac981a7a9c5c3603cf1c6bfbb0ad9cdce1a09b27c9f64270c9688916d50"
Oct 30 23:00:35.408100 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.408031    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5d31ac981a7a9c5c3603cf1c6bfbb0ad9cdce1a09b27c9f64270c9688916d50"} err="failed to get container status \"a5d31ac981a7a9c5c3603cf1c6bfbb0ad9cdce1a09b27c9f64270c9688916d50\": rpc error: code = NotFound desc = could not find container \"a5d31ac981a7a9c5c3603cf1c6bfbb0ad9cdce1a09b27c9f64270c9688916d50\": container with ID starting with a5d31ac981a7a9c5c3603cf1c6bfbb0ad9cdce1a09b27c9f64270c9688916d50 not found: ID does not exist"
Oct 30 23:00:35.412399 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.412371    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-d97b6f8b7-p8h4c"]
Oct 30 23:00:35.449170 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.449088    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9"]
Oct 30 23:00:35.453007 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:35.452977    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76d54a-5c57cb5785-lvdp9"]
Oct 30 23:00:36.541133 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:36.541075    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:36.777248 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:36.777188    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1ea4fda-c921-46a1-8107-441bea92acf1" path="/var/lib/kubelet/pods/b1ea4fda-c921-46a1-8107-441bea92acf1/volumes"
Oct 30 23:00:36.777696 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:36.777681    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb8af940-2412-4e39-a599-450debc2e114" path="/var/lib/kubelet/pods/cb8af940-2412-4e39-a599-450debc2e114/volumes"
Oct 30 23:00:38.379043 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:38.379005    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:39.652559 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:39.652508    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt" podUID="9939f760-86ef-484a-876c-6495d528af48" containerName="payload-processing" containerID="cri-o://47ebd65232f113b89686bc9294e80f7ef13c592c276d9bf548eb601ec244bb34" gracePeriod=25
Oct 30 23:00:39.961706 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:39.961673    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt"
Oct 30 23:00:40.064255 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:40.063924    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9939f760-86ef-484a-876c-6495d528af48-kube-api-access-8t6g8\" (UniqueName: \"kubernetes.io/projected/9939f760-86ef-484a-876c-6495d528af48-kube-api-access-8t6g8\") pod \"9939f760-86ef-484a-876c-6495d528af48\" (UID: \"9939f760-86ef-484a-876c-6495d528af48\") "
Oct 30 23:00:40.064255 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:40.064074    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9939f760-86ef-484a-876c-6495d528af48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9939f760-86ef-484a-876c-6495d528af48-plugins-config-volume\") pod \"9939f760-86ef-484a-876c-6495d528af48\" (UID: \"9939f760-86ef-484a-876c-6495d528af48\") "
Oct 30 23:00:40.066143 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:40.066112    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9939f760-86ef-484a-876c-6495d528af48-plugins-config-volume" pod "9939f760-86ef-484a-876c-6495d528af48" (UID: "9939f760-86ef-484a-876c-6495d528af48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:40.070272 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:40.069241    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9939f760-86ef-484a-876c-6495d528af48-kube-api-access-8t6g8" pod "9939f760-86ef-484a-876c-6495d528af48" (UID: "9939f760-86ef-484a-876c-6495d528af48"). InnerVolumeSpecName "kube-api-access-8t6g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:40.122292 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:40.122211    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:40.167476 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:40.167440    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t6g8\" (UniqueName: \"kubernetes.io/projected/9939f760-86ef-484a-876c-6495d528af48-kube-api-access-8t6g8\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:00:40.167476 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:40.167472    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9939f760-86ef-484a-876c-6495d528af48-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:00:40.305499 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:00:40.305459    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:00:40.400175 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:40.400137    4889 generic.go:356] "Generic (PLEG): container finished" podID="9939f760-86ef-484a-876c-6495d528af48" containerID="47ebd65232f113b89686bc9294e80f7ef13c592c276d9bf548eb601ec244bb34" exitCode=0
Oct 30 23:00:40.400417 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:40.400197    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt"
Oct 30 23:00:40.400417 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:40.400205    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt" event={"ID":"9939f760-86ef-484a-876c-6495d528af48","Type":"ContainerDied","Data":"47ebd65232f113b89686bc9294e80f7ef13c592c276d9bf548eb601ec244bb34"}
Oct 30 23:00:40.400417 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:40.400332    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt" event={"ID":"9939f760-86ef-484a-876c-6495d528af48","Type":"ContainerDied","Data":"d4f48759f5813d97416f2b27c98f757a6bca16ac94443031164ad4be4cf75b07"}
Oct 30 23:00:40.400417 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:40.400366    4889 scope.go:122] "RemoveContainer" containerID="47ebd65232f113b89686bc9294e80f7ef13c592c276d9bf548eb601ec244bb34"
Oct 30 23:00:40.413085 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:40.413054    4889 scope.go:122] "RemoveContainer" containerID="47ebd65232f113b89686bc9294e80f7ef13c592c276d9bf548eb601ec244bb34"
Oct 30 23:00:40.413467 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:00:40.413445    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47ebd65232f113b89686bc9294e80f7ef13c592c276d9bf548eb601ec244bb34\": container with ID starting with 47ebd65232f113b89686bc9294e80f7ef13c592c276d9bf548eb601ec244bb34 not found: ID does not exist" containerID="47ebd65232f113b89686bc9294e80f7ef13c592c276d9bf548eb601ec244bb34"
Oct 30 23:00:40.413543 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:40.413478    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47ebd65232f113b89686bc9294e80f7ef13c592c276d9bf548eb601ec244bb34"} err="failed to get container status \"47ebd65232f113b89686bc9294e80f7ef13c592c276d9bf548eb601ec244bb34\": rpc error: code = NotFound desc = could not find container \"47ebd65232f113b89686bc9294e80f7ef13c592c276d9bf548eb601ec244bb34\": container with ID starting with 47ebd65232f113b89686bc9294e80f7ef13c592c276d9bf548eb601ec244bb34 not found: ID does not exist"
Oct 30 23:00:40.462709 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:40.462666    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt"]
Oct 30 23:00:40.475485 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:40.475448    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f94561-5ddf9d6659-rntqt"]
Oct 30 23:00:40.777051 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:40.777015    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9939f760-86ef-484a-876c-6495d528af48" path="/var/lib/kubelet/pods/9939f760-86ef-484a-876c-6495d528af48/volumes"
Oct 30 23:00:41.897284 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:41.893858    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:43.820913 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:43.820877    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:45.175260 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:45.175138    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:45.368829 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:45.368783    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:45.731203 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:45.731155    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:46.464801 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:46.464763    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:47.726201 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:47.726157    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:49.755177 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:49.755137    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:51.366987 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:51.366950    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:51.707170 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:51.707067    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:53.037289 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:53.037252    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp"]
Oct 30 23:00:53.238691 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:53.238650    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:53.417627 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:53.417317    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:53.601351 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:53.601292    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:55.874104 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:55.874062    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:56.147316 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:56.146442    4889 patch_prober.go:29] interesting pod/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": read tcp 10.132.0.2:60336->10.132.0.239:15021: read: connection reset by peer" start-of-body=
Oct 30 23:00:56.147316 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:56.146528    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh" podUID="df4966ae-49e6-461e-b677-6111da780e93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": read tcp 10.132.0.2:60336->10.132.0.239:15021: read: connection reset by peer"
Oct 30 23:00:56.601899 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:56.601862    4889 generic.go:356] "Generic (PLEG): container finished" podID="df4966ae-49e6-461e-b677-6111da780e93" containerID="2f9d5c5f3c357c6ed6f1e5fe72ee40e13646424e2f2b8dd53d6cb0e392d8d665" exitCode=0
Oct 30 23:00:56.602115 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:56.601930    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh" event={"ID":"df4966ae-49e6-461e-b677-6111da780e93","Type":"ContainerDied","Data":"2f9d5c5f3c357c6ed6f1e5fe72ee40e13646424e2f2b8dd53d6cb0e392d8d665"}
Oct 30 23:00:56.602425 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:56.602410    4889 scope.go:122] "RemoveContainer" containerID="2f9d5c5f3c357c6ed6f1e5fe72ee40e13646424e2f2b8dd53d6cb0e392d8d665"
Oct 30 23:00:56.604262 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:56.604215    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:00:56.604343 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:56.604300    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:00:56.604343 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:56.604330    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:00:57.622316 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:57.613426    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh" event={"ID":"df4966ae-49e6-461e-b677-6111da780e93","Type":"ContainerStarted","Data":"7ddc1f3e950b2fdc2698a2862187414e6680d315a0b97a0f70da00a745880367"}
Oct 30 23:00:57.748753 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:57.745991    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:58.034520 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:58.034472    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 23:00:58.034872 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:58.034844    4889 patch_prober.go:29] interesting pod/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 23:00:58.034994 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:58.034901    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh" podUID="df4966ae-49e6-461e-b677-6111da780e93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 23:00:58.038397 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:58.038351    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp" podUID="3e35e8fe-0081-41ff-851f-c61c25c3942a" containerName="payload-processing" containerID="cri-o://74bd7031b4b5c117bfd0bf7f80654127e56552c6bbb3df532e280c2d201595fb" gracePeriod=25
Oct 30 23:00:58.386552 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:58.386525    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp"
Oct 30 23:00:58.495540 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:58.495502    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e35e8fe-0081-41ff-851f-c61c25c3942a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e35e8fe-0081-41ff-851f-c61c25c3942a-plugins-config-volume\") pod \"3e35e8fe-0081-41ff-851f-c61c25c3942a\" (UID: \"3e35e8fe-0081-41ff-851f-c61c25c3942a\") "
Oct 30 23:00:58.495784 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:58.495590    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e35e8fe-0081-41ff-851f-c61c25c3942a-kube-api-access-qrlwq\" (UniqueName: \"kubernetes.io/projected/3e35e8fe-0081-41ff-851f-c61c25c3942a-kube-api-access-qrlwq\") pod \"3e35e8fe-0081-41ff-851f-c61c25c3942a\" (UID: \"3e35e8fe-0081-41ff-851f-c61c25c3942a\") "
Oct 30 23:00:58.496030 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:58.495996    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e35e8fe-0081-41ff-851f-c61c25c3942a-plugins-config-volume" pod "3e35e8fe-0081-41ff-851f-c61c25c3942a" (UID: "3e35e8fe-0081-41ff-851f-c61c25c3942a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:58.496677 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:58.496652    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e35e8fe-0081-41ff-851f-c61c25c3942a-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:00:58.498328 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:58.498295    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e35e8fe-0081-41ff-851f-c61c25c3942a-kube-api-access-qrlwq" pod "3e35e8fe-0081-41ff-851f-c61c25c3942a" (UID: "3e35e8fe-0081-41ff-851f-c61c25c3942a"). InnerVolumeSpecName "kube-api-access-qrlwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:58.598412 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:58.598368    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrlwq\" (UniqueName: \"kubernetes.io/projected/3e35e8fe-0081-41ff-851f-c61c25c3942a-kube-api-access-qrlwq\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:00:58.642283 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:58.642169    4889 generic.go:356] "Generic (PLEG): container finished" podID="3e35e8fe-0081-41ff-851f-c61c25c3942a" containerID="74bd7031b4b5c117bfd0bf7f80654127e56552c6bbb3df532e280c2d201595fb" exitCode=0
Oct 30 23:00:58.642283 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:58.642217    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp"
Oct 30 23:00:58.642283 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:58.642262    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp" event={"ID":"3e35e8fe-0081-41ff-851f-c61c25c3942a","Type":"ContainerDied","Data":"74bd7031b4b5c117bfd0bf7f80654127e56552c6bbb3df532e280c2d201595fb"}
Oct 30 23:00:58.642873 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:58.642325    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp" event={"ID":"3e35e8fe-0081-41ff-851f-c61c25c3942a","Type":"ContainerDied","Data":"7a17e7123c6801156728d6af1f355e50ea977c621622a146958d1896611ffd56"}
Oct 30 23:00:58.642873 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:58.642353    4889 scope.go:122] "RemoveContainer" containerID="74bd7031b4b5c117bfd0bf7f80654127e56552c6bbb3df532e280c2d201595fb"
Oct 30 23:00:58.656068 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:58.656043    4889 scope.go:122] "RemoveContainer" containerID="74bd7031b4b5c117bfd0bf7f80654127e56552c6bbb3df532e280c2d201595fb"
Oct 30 23:00:58.656417 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:00:58.656399    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74bd7031b4b5c117bfd0bf7f80654127e56552c6bbb3df532e280c2d201595fb\": container with ID starting with 74bd7031b4b5c117bfd0bf7f80654127e56552c6bbb3df532e280c2d201595fb not found: ID does not exist" containerID="74bd7031b4b5c117bfd0bf7f80654127e56552c6bbb3df532e280c2d201595fb"
Oct 30 23:00:58.656492 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:58.656426    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74bd7031b4b5c117bfd0bf7f80654127e56552c6bbb3df532e280c2d201595fb"} err="failed to get container status \"74bd7031b4b5c117bfd0bf7f80654127e56552c6bbb3df532e280c2d201595fb\": rpc error: code = NotFound desc = could not find container \"74bd7031b4b5c117bfd0bf7f80654127e56552c6bbb3df532e280c2d201595fb\": container with ID starting with 74bd7031b4b5c117bfd0bf7f80654127e56552c6bbb3df532e280c2d201595fb not found: ID does not exist"
Oct 30 23:00:58.706653 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:58.706617    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp"]
Oct 30 23:00:58.731251 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:58.721157    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-8f44c5dc9-jgtzp"]
Oct 30 23:00:58.780759 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:58.779648    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e35e8fe-0081-41ff-851f-c61c25c3942a" path="/var/lib/kubelet/pods/3e35e8fe-0081-41ff-851f-c61c25c3942a/volumes"
Oct 30 23:00:58.862189 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:58.862145    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:59.032247 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:59.031407    4889 patch_prober.go:29] interesting pod/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 23:00:59.032247 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:59.031463    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh" podUID="df4966ae-49e6-461e-b677-6111da780e93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 23:00:59.782197 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:00:59.782152    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:00.034778 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:00.034683    4889 patch_prober.go:29] interesting pod/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 23:01:00.034778 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:00.034753    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh" podUID="df4966ae-49e6-461e-b677-6111da780e93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 23:01:00.631253 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:00.629283    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:01.032770 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:01.032727    4889 patch_prober.go:29] interesting pod/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 23:01:01.033478 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:01.033441    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh" podUID="df4966ae-49e6-461e-b677-6111da780e93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 23:01:01.574571 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:01.574526    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:02.037823 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:02.037760    4889 patch_prober.go:29] interesting pod/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 23:01:02.038513 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:02.038476    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh" podUID="df4966ae-49e6-461e-b677-6111da780e93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 23:01:03.079508 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:03.079468    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 23:01:03.080406 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:03.080375    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 23:01:03.081761 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:03.081739    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 23:01:03.335015 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:03.334910    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:05.123750 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:05.123708    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:06.985095 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:06.985050    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:07.152675 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:07.152610    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:07.342939 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:07.342421    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:10.306077 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:01:10.306034    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:01:10.981123 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:10.981075    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:11.653244 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:11.653200    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:13.573477 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:13.573434    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:15.012841 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:15.012807    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:15.181394 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:15.181354    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:16.164246 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:16.162567    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:17.945507 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:17.945452    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:19.394883 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:19.394841    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"]
Oct 30 23:01:19.493117 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:19.493067    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:19.782739 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:19.782704    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:21.010451 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.010410    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:21.196111 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.196069    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:21.499675 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.499626    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2"]
Oct 30 23:01:21.500627 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500598    4889 cpu_manager.go:414] "containerMap: removing container" podUID="c9a31620-90fb-478e-b32a-f34d8ab55384" containerName="cleanup"
Oct 30 23:01:21.500627 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500627    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9a31620-90fb-478e-b32a-f34d8ab55384" containerName="cleanup"
Oct 30 23:01:21.500909 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500650    4889 cpu_manager.go:414] "containerMap: removing container" podUID="73fcb5fd-f6ef-4313-a95e-993ca08672b9" containerName="cleanup"
Oct 30 23:01:21.500909 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500658    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73fcb5fd-f6ef-4313-a95e-993ca08672b9" containerName="cleanup"
Oct 30 23:01:21.500909 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500670    4889 cpu_manager.go:414] "containerMap: removing container" podUID="69be2b84-9d09-48b8-876e-67f49f56d9a6" containerName="cleanup"
Oct 30 23:01:21.500909 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500677    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69be2b84-9d09-48b8-876e-67f49f56d9a6" containerName="cleanup"
Oct 30 23:01:21.500909 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500700    4889 cpu_manager.go:414] "containerMap: removing container" podUID="83c059bc-93eb-4c5e-9803-098ac14602ad" containerName="cleanup"
Oct 30 23:01:21.500909 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500706    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c059bc-93eb-4c5e-9803-098ac14602ad" containerName="cleanup"
Oct 30 23:01:21.500909 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500717    4889 cpu_manager.go:414] "containerMap: removing container" podUID="73fcb5fd-f6ef-4313-a95e-993ca08672b9" containerName="cleanup"
Oct 30 23:01:21.500909 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500723    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73fcb5fd-f6ef-4313-a95e-993ca08672b9" containerName="cleanup"
Oct 30 23:01:21.500909 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500736    4889 cpu_manager.go:414] "containerMap: removing container" podUID="83c059bc-93eb-4c5e-9803-098ac14602ad" containerName="cleanup"
Oct 30 23:01:21.500909 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500741    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c059bc-93eb-4c5e-9803-098ac14602ad" containerName="cleanup"
Oct 30 23:01:21.500909 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500756    4889 cpu_manager.go:414] "containerMap: removing container" podUID="9939f760-86ef-484a-876c-6495d528af48" containerName="payload-processing"
Oct 30 23:01:21.500909 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500763    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9939f760-86ef-484a-876c-6495d528af48" containerName="payload-processing"
Oct 30 23:01:21.500909 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500772    4889 cpu_manager.go:414] "containerMap: removing container" podUID="c9a31620-90fb-478e-b32a-f34d8ab55384" containerName="cleanup"
Oct 30 23:01:21.500909 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500778    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9a31620-90fb-478e-b32a-f34d8ab55384" containerName="cleanup"
Oct 30 23:01:21.500909 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500795    4889 cpu_manager.go:414] "containerMap: removing container" podUID="b89aa2e2-f770-4bb7-903e-f3c2b4878359" containerName="cleanup"
Oct 30 23:01:21.500909 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500802    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b89aa2e2-f770-4bb7-903e-f3c2b4878359" containerName="cleanup"
Oct 30 23:01:21.500909 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500813    4889 cpu_manager.go:414] "containerMap: removing container" podUID="4fe9cad0-baac-45db-a8e4-63a6e83d00c0" containerName="cleanup"
Oct 30 23:01:21.500909 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500821    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe9cad0-baac-45db-a8e4-63a6e83d00c0" containerName="cleanup"
Oct 30 23:01:21.500909 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500849    4889 cpu_manager.go:414] "containerMap: removing container" podUID="56702f85-adba-4f54-a21d-1f4ef1ad4987" containerName="cleanup"
Oct 30 23:01:21.500909 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500857    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56702f85-adba-4f54-a21d-1f4ef1ad4987" containerName="cleanup"
Oct 30 23:01:21.500909 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500869    4889 cpu_manager.go:414] "containerMap: removing container" podUID="b1ea4fda-c921-46a1-8107-441bea92acf1" containerName="payload-pre-processing"
Oct 30 23:01:21.500909 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500876    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1ea4fda-c921-46a1-8107-441bea92acf1" containerName="payload-pre-processing"
Oct 30 23:01:21.500909 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500894    4889 cpu_manager.go:414] "containerMap: removing container" podUID="cb8af940-2412-4e39-a599-450debc2e114" containerName="payload-processing"
Oct 30 23:01:21.500909 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500901    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb8af940-2412-4e39-a599-450debc2e114" containerName="payload-processing"
Oct 30 23:01:21.500909 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500916    4889 cpu_manager.go:414] "containerMap: removing container" podUID="b89aa2e2-f770-4bb7-903e-f3c2b4878359" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500924    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b89aa2e2-f770-4bb7-903e-f3c2b4878359" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500933    4889 cpu_manager.go:414] "containerMap: removing container" podUID="b89aa2e2-f770-4bb7-903e-f3c2b4878359" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500940    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b89aa2e2-f770-4bb7-903e-f3c2b4878359" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500959    4889 cpu_manager.go:414] "containerMap: removing container" podUID="3e35e8fe-0081-41ff-851f-c61c25c3942a" containerName="payload-processing"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500967    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e35e8fe-0081-41ff-851f-c61c25c3942a" containerName="payload-processing"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500976    4889 cpu_manager.go:414] "containerMap: removing container" podUID="83c059bc-93eb-4c5e-9803-098ac14602ad" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500983    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c059bc-93eb-4c5e-9803-098ac14602ad" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.500995    4889 cpu_manager.go:414] "containerMap: removing container" podUID="56702f85-adba-4f54-a21d-1f4ef1ad4987" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501002    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56702f85-adba-4f54-a21d-1f4ef1ad4987" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501040    4889 cpu_manager.go:414] "containerMap: removing container" podUID="56702f85-adba-4f54-a21d-1f4ef1ad4987" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501050    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56702f85-adba-4f54-a21d-1f4ef1ad4987" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501063    4889 cpu_manager.go:414] "containerMap: removing container" podUID="69be2b84-9d09-48b8-876e-67f49f56d9a6" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501070    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69be2b84-9d09-48b8-876e-67f49f56d9a6" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501088    4889 cpu_manager.go:414] "containerMap: removing container" podUID="4fe9cad0-baac-45db-a8e4-63a6e83d00c0" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501095    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe9cad0-baac-45db-a8e4-63a6e83d00c0" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501312    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="56702f85-adba-4f54-a21d-1f4ef1ad4987" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501331    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1ea4fda-c921-46a1-8107-441bea92acf1" containerName="payload-pre-processing"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501345    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb8af940-2412-4e39-a599-450debc2e114" containerName="payload-processing"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501358    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="b89aa2e2-f770-4bb7-903e-f3c2b4878359" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501367    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="b89aa2e2-f770-4bb7-903e-f3c2b4878359" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501377    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe9cad0-baac-45db-a8e4-63a6e83d00c0" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501386    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="69be2b84-9d09-48b8-876e-67f49f56d9a6" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501395    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e35e8fe-0081-41ff-851f-c61c25c3942a" containerName="payload-processing"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501403    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="83c059bc-93eb-4c5e-9803-098ac14602ad" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501412    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="56702f85-adba-4f54-a21d-1f4ef1ad4987" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501422    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9a31620-90fb-478e-b32a-f34d8ab55384" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501435    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="56702f85-adba-4f54-a21d-1f4ef1ad4987" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501443    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="69be2b84-9d09-48b8-876e-67f49f56d9a6" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501456    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe9cad0-baac-45db-a8e4-63a6e83d00c0" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501466    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9a31620-90fb-478e-b32a-f34d8ab55384" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501476    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="73fcb5fd-f6ef-4313-a95e-993ca08672b9" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501487    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="69be2b84-9d09-48b8-876e-67f49f56d9a6" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501498    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="73fcb5fd-f6ef-4313-a95e-993ca08672b9" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501508    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="73fcb5fd-f6ef-4313-a95e-993ca08672b9" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501515    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="83c059bc-93eb-4c5e-9803-098ac14602ad" containerName="cleanup"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501523    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="9939f760-86ef-484a-876c-6495d528af48" containerName="payload-processing"
Oct 30 23:01:21.502560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.501534    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9a31620-90fb-478e-b32a-f34d8ab55384" containerName="cleanup"
Oct 30 23:01:21.505900 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.505875    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2"
Oct 30 23:01:21.540263 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.538288    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f94561\""
Oct 30 23:01:21.540263 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.538442    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f94561-dockercfg-nh4xn\""
Oct 30 23:01:21.576386 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.576277    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2"]
Oct 30 23:01:21.650203 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.650151    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428b38e7-52ac-4f32-a8be-ea74f109c4d1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2\" (UID: \"428b38e7-52ac-4f32-a8be-ea74f109c4d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2"
Oct 30 23:01:21.650699 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.650667    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmg42\" (UniqueName: \"kubernetes.io/projected/428b38e7-52ac-4f32-a8be-ea74f109c4d1-kube-api-access-qmg42\") pod \"payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2\" (UID: \"428b38e7-52ac-4f32-a8be-ea74f109c4d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2"
Oct 30 23:01:21.751910 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.751799    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428b38e7-52ac-4f32-a8be-ea74f109c4d1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2\" (UID: \"428b38e7-52ac-4f32-a8be-ea74f109c4d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2"
Oct 30 23:01:21.752108 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.752019    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmg42\" (UniqueName: \"kubernetes.io/projected/428b38e7-52ac-4f32-a8be-ea74f109c4d1-kube-api-access-qmg42\") pod \"payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2\" (UID: \"428b38e7-52ac-4f32-a8be-ea74f109c4d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2"
Oct 30 23:01:21.752677 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.752652    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428b38e7-52ac-4f32-a8be-ea74f109c4d1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2\" (UID: \"428b38e7-52ac-4f32-a8be-ea74f109c4d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2"
Oct 30 23:01:21.773739 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.773699    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmg42\" (UniqueName: \"kubernetes.io/projected/428b38e7-52ac-4f32-a8be-ea74f109c4d1-kube-api-access-qmg42\") pod \"payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2\" (UID: \"428b38e7-52ac-4f32-a8be-ea74f109c4d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2"
Oct 30 23:01:21.875067 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:21.875015    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2"
Oct 30 23:01:22.132083 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:22.131968    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:22.186983 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:22.186950    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2"]
Oct 30 23:01:22.187152 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:01:22.187027    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428b38e7_52ac_4f32_a8be_ea74f109c4d1.slice/crio-66abc1da68bbb22ad5b8cc4f4a426c4657add692ca6a640b7eda0d03e51babaa WatchSource:0}: Error finding container 66abc1da68bbb22ad5b8cc4f4a426c4657add692ca6a640b7eda0d03e51babaa: Status 404 returned error can't find the container with id 66abc1da68bbb22ad5b8cc4f4a426c4657add692ca6a640b7eda0d03e51babaa
Oct 30 23:01:22.848994 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:22.848948    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2" event={"ID":"428b38e7-52ac-4f32-a8be-ea74f109c4d1","Type":"ContainerStarted","Data":"8d9d49f2df099fdd27f0e79993b61396725a2b85157663cff4e9991f7eebab72"}
Oct 30 23:01:22.849207 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:22.849019    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2" event={"ID":"428b38e7-52ac-4f32-a8be-ea74f109c4d1","Type":"ContainerStarted","Data":"66abc1da68bbb22ad5b8cc4f4a426c4657add692ca6a640b7eda0d03e51babaa"}
Oct 30 23:01:22.849207 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:22.849050    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2"
Oct 30 23:01:22.898262 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:22.898174    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2" podStartSLOduration=1.8981494300000001 podStartE2EDuration="1.89814943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:01:22.892498294 +0000 UTC m=+3268.248864611" watchObservedRunningTime="2026-10-30 23:01:22.89814943 +0000 UTC m=+3268.254515745"
Oct 30 23:01:23.917994 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:23.917954    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:25.739203 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:25.739163    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:27.684025 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:27.683988    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:29.388366 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:29.388329    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:31.181572 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:31.181529    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:32.946582 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:32.946530    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:33.860825 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:33.860788    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2"
Oct 30 23:01:34.424681 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.424594    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf" podUID="2eb013be-cbf1-4d26-9393-00c6498157de" containerName="main" containerID="cri-o://365ca75a9130796f64070d9ff210f19019fd80f11a753f9bb32fe6e4bca1e8bb" gracePeriod=45
Oct 30 23:01:34.726638 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.726608    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"
Oct 30 23:01:34.846819 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.846787    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:34.854020 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.853988    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-home\" (UniqueName: \"kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-home\") pod \"2eb013be-cbf1-4d26-9393-00c6498157de\" (UID: \"2eb013be-cbf1-4d26-9393-00c6498157de\") "
Oct 30 23:01:34.854134 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.854057    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2eb013be-cbf1-4d26-9393-00c6498157de-tls-certs\" (UniqueName: \"kubernetes.io/secret/2eb013be-cbf1-4d26-9393-00c6498157de-tls-certs\") pod \"2eb013be-cbf1-4d26-9393-00c6498157de\" (UID: \"2eb013be-cbf1-4d26-9393-00c6498157de\") "
Oct 30 23:01:34.854134 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.854078    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-dshm\") pod \"2eb013be-cbf1-4d26-9393-00c6498157de\" (UID: \"2eb013be-cbf1-4d26-9393-00c6498157de\") "
Oct 30 23:01:34.854327 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.854302    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-home" pod "2eb013be-cbf1-4d26-9393-00c6498157de" (UID: "2eb013be-cbf1-4d26-9393-00c6498157de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:01:34.854404 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.854309    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-tmp-dir\") pod \"2eb013be-cbf1-4d26-9393-00c6498157de\" (UID: \"2eb013be-cbf1-4d26-9393-00c6498157de\") "
Oct 30 23:01:34.854404 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.854390    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-model-cache\") pod \"2eb013be-cbf1-4d26-9393-00c6498157de\" (UID: \"2eb013be-cbf1-4d26-9393-00c6498157de\") "
Oct 30 23:01:34.854518 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.854413    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eb013be-cbf1-4d26-9393-00c6498157de-kube-api-access-4qgrt\" (UniqueName: \"kubernetes.io/projected/2eb013be-cbf1-4d26-9393-00c6498157de-kube-api-access-4qgrt\") pod \"2eb013be-cbf1-4d26-9393-00c6498157de\" (UID: \"2eb013be-cbf1-4d26-9393-00c6498157de\") "
Oct 30 23:01:34.854573 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.854506    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-tmp-dir" pod "2eb013be-cbf1-4d26-9393-00c6498157de" (UID: "2eb013be-cbf1-4d26-9393-00c6498157de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:01:34.854715 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.854691    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-model-cache" pod "2eb013be-cbf1-4d26-9393-00c6498157de" (UID: "2eb013be-cbf1-4d26-9393-00c6498157de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:01:34.855401 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.855377    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:01:34.855401 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.855401    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:01:34.856125 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.855415    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:01:34.856714 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.856688    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-dshm" pod "2eb013be-cbf1-4d26-9393-00c6498157de" (UID: "2eb013be-cbf1-4d26-9393-00c6498157de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:01:34.856820 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.856750    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eb013be-cbf1-4d26-9393-00c6498157de-tls-certs" pod "2eb013be-cbf1-4d26-9393-00c6498157de" (UID: "2eb013be-cbf1-4d26-9393-00c6498157de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:01:34.857292 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.857218    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb013be-cbf1-4d26-9393-00c6498157de-kube-api-access-4qgrt" pod "2eb013be-cbf1-4d26-9393-00c6498157de" (UID: "2eb013be-cbf1-4d26-9393-00c6498157de"). InnerVolumeSpecName "kube-api-access-4qgrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:01:34.956965 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.956847    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qgrt\" (UniqueName: \"kubernetes.io/projected/2eb013be-cbf1-4d26-9393-00c6498157de-kube-api-access-4qgrt\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:01:34.956965 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.956881    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eb013be-cbf1-4d26-9393-00c6498157de-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:01:34.956965 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.956893    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eb013be-cbf1-4d26-9393-00c6498157de-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:01:34.960527 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.960489    4889 generic.go:356] "Generic (PLEG): container finished" podID="2eb013be-cbf1-4d26-9393-00c6498157de" containerID="365ca75a9130796f64070d9ff210f19019fd80f11a753f9bb32fe6e4bca1e8bb" exitCode=0
Oct 30 23:01:34.960701 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.960588    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"
Oct 30 23:01:34.960701 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.960615    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf" event={"ID":"2eb013be-cbf1-4d26-9393-00c6498157de","Type":"ContainerDied","Data":"365ca75a9130796f64070d9ff210f19019fd80f11a753f9bb32fe6e4bca1e8bb"}
Oct 30 23:01:34.960701 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.960672    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf" event={"ID":"2eb013be-cbf1-4d26-9393-00c6498157de","Type":"ContainerDied","Data":"05839fcfc62313ea2a3946d3b91c82abfff70831f1dc97aa8398ce71f8048aa5"}
Oct 30 23:01:34.960701 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.960695    4889 scope.go:122] "RemoveContainer" containerID="365ca75a9130796f64070d9ff210f19019fd80f11a753f9bb32fe6e4bca1e8bb"
Oct 30 23:01:34.975936 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.975910    4889 scope.go:122] "RemoveContainer" containerID="365ca75a9130796f64070d9ff210f19019fd80f11a753f9bb32fe6e4bca1e8bb"
Oct 30 23:01:34.976465 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:01:34.976440    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"365ca75a9130796f64070d9ff210f19019fd80f11a753f9bb32fe6e4bca1e8bb\": container with ID starting with 365ca75a9130796f64070d9ff210f19019fd80f11a753f9bb32fe6e4bca1e8bb not found: ID does not exist" containerID="365ca75a9130796f64070d9ff210f19019fd80f11a753f9bb32fe6e4bca1e8bb"
Oct 30 23:01:34.976600 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:34.976473    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"365ca75a9130796f64070d9ff210f19019fd80f11a753f9bb32fe6e4bca1e8bb"} err="failed to get container status \"365ca75a9130796f64070d9ff210f19019fd80f11a753f9bb32fe6e4bca1e8bb\": rpc error: code = NotFound desc = could not find container \"365ca75a9130796f64070d9ff210f19019fd80f11a753f9bb32fe6e4bca1e8bb\": container with ID starting with 365ca75a9130796f64070d9ff210f19019fd80f11a753f9bb32fe6e4bca1e8bb not found: ID does not exist"
Oct 30 23:01:35.020426 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:35.020390    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"]
Oct 30 23:01:35.024511 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:35.024477    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-6f2eba/rate-test-model-6f2eba-kserve-855bb4998f-bprqf"]
Oct 30 23:01:36.154323 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.154282    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:36.340301 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.340214    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:36.673748 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.673709    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf"]
Oct 30 23:01:36.674528 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.674508    4889 cpu_manager.go:414] "containerMap: removing container" podUID="2eb013be-cbf1-4d26-9393-00c6498157de" containerName="main"
Oct 30 23:01:36.674599 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.674529    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eb013be-cbf1-4d26-9393-00c6498157de" containerName="main"
Oct 30 23:01:36.674599 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.674591    4889 cpu_manager.go:414] "containerMap: removing container" podUID="69be2b84-9d09-48b8-876e-67f49f56d9a6" containerName="cleanup"
Oct 30 23:01:36.674599 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.674597    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69be2b84-9d09-48b8-876e-67f49f56d9a6" containerName="cleanup"
Oct 30 23:01:36.674715 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.674617    4889 cpu_manager.go:414] "containerMap: removing container" podUID="73fcb5fd-f6ef-4313-a95e-993ca08672b9" containerName="cleanup"
Oct 30 23:01:36.674715 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.674623    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73fcb5fd-f6ef-4313-a95e-993ca08672b9" containerName="cleanup"
Oct 30 23:01:36.674715 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.674631    4889 cpu_manager.go:414] "containerMap: removing container" podUID="c9a31620-90fb-478e-b32a-f34d8ab55384" containerName="cleanup"
Oct 30 23:01:36.674715 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.674636    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9a31620-90fb-478e-b32a-f34d8ab55384" containerName="cleanup"
Oct 30 23:01:36.674715 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.674648    4889 cpu_manager.go:414] "containerMap: removing container" podUID="4fe9cad0-baac-45db-a8e4-63a6e83d00c0" containerName="cleanup"
Oct 30 23:01:36.674715 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.674652    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe9cad0-baac-45db-a8e4-63a6e83d00c0" containerName="cleanup"
Oct 30 23:01:36.674918 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.674759    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="83c059bc-93eb-4c5e-9803-098ac14602ad" containerName="cleanup"
Oct 30 23:01:36.674918 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.674771    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eb013be-cbf1-4d26-9393-00c6498157de" containerName="main"
Oct 30 23:01:36.674918 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.674779    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="b89aa2e2-f770-4bb7-903e-f3c2b4878359" containerName="cleanup"
Oct 30 23:01:36.674918 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.674787    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe9cad0-baac-45db-a8e4-63a6e83d00c0" containerName="cleanup"
Oct 30 23:01:36.680035 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.680005    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf"
Oct 30 23:01:36.698635 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.698600    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf"]
Oct 30 23:01:36.707549 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.706258    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-cdfa86\""
Oct 30 23:01:36.707549 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.706274    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-cdfa86-dockercfg-xpvmj\""
Oct 30 23:01:36.776812 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.776777    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eb013be-cbf1-4d26-9393-00c6498157de" path="/var/lib/kubelet/pods/2eb013be-cbf1-4d26-9393-00c6498157de/volumes"
Oct 30 23:01:36.795466 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.795431    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjhsl\" (UniqueName: \"kubernetes.io/projected/c27dca2f-2be7-4a83-bdc0-9df7082f0027-kube-api-access-pjhsl\") pod \"payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf\" (UID: \"c27dca2f-2be7-4a83-bdc0-9df7082f0027\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf"
Oct 30 23:01:36.795677 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.795592    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c27dca2f-2be7-4a83-bdc0-9df7082f0027-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf\" (UID: \"c27dca2f-2be7-4a83-bdc0-9df7082f0027\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf"
Oct 30 23:01:36.896891 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.896840    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjhsl\" (UniqueName: \"kubernetes.io/projected/c27dca2f-2be7-4a83-bdc0-9df7082f0027-kube-api-access-pjhsl\") pod \"payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf\" (UID: \"c27dca2f-2be7-4a83-bdc0-9df7082f0027\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf"
Oct 30 23:01:36.897113 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.897033    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c27dca2f-2be7-4a83-bdc0-9df7082f0027-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf\" (UID: \"c27dca2f-2be7-4a83-bdc0-9df7082f0027\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf"
Oct 30 23:01:36.897792 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.897765    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c27dca2f-2be7-4a83-bdc0-9df7082f0027-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf\" (UID: \"c27dca2f-2be7-4a83-bdc0-9df7082f0027\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf"
Oct 30 23:01:36.917764 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.917728    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjhsl\" (UniqueName: \"kubernetes.io/projected/c27dca2f-2be7-4a83-bdc0-9df7082f0027-kube-api-access-pjhsl\") pod \"payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf\" (UID: \"c27dca2f-2be7-4a83-bdc0-9df7082f0027\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf"
Oct 30 23:01:37.000029 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:36.999987    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf"
Oct 30 23:01:37.323666 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:37.323633    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf"]
Oct 30 23:01:37.324031 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:01:37.323684    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27dca2f_2be7_4a83_bdc0_9df7082f0027.slice/crio-90b0ab90d45cecfa015dda53a898b4bc7d408eae3301b392350b5b734873c405 WatchSource:0}: Error finding container 90b0ab90d45cecfa015dda53a898b4bc7d408eae3301b392350b5b734873c405: Status 404 returned error can't find the container with id 90b0ab90d45cecfa015dda53a898b4bc7d408eae3301b392350b5b734873c405
Oct 30 23:01:37.338529 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:37.338486    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:38.004036 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:38.003995    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf" event={"ID":"c27dca2f-2be7-4a83-bdc0-9df7082f0027","Type":"ContainerStarted","Data":"1dcc57b08841bea7db75a2463a5430763d2a3418aca74e6737ba21eaed02d97d"}
Oct 30 23:01:38.004270 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:38.004103    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf"
Oct 30 23:01:38.004270 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:38.004126    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf" event={"ID":"c27dca2f-2be7-4a83-bdc0-9df7082f0027","Type":"ContainerStarted","Data":"90b0ab90d45cecfa015dda53a898b4bc7d408eae3301b392350b5b734873c405"}
Oct 30 23:01:38.050581 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:38.050514    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf" podStartSLOduration=2.050496464 podStartE2EDuration="2.050496464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:01:38.048448765 +0000 UTC m=+3283.404815084" watchObservedRunningTime="2026-10-30 23:01:38.050496464 +0000 UTC m=+3283.406862781"
Oct 30 23:01:38.303410 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:38.303317    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:38.589309 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:38.581263    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:38.910107 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:38.909982    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:39.150410 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:39.150359    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:40.306205 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:01:40.306167    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:01:40.949701 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:40.949651    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:42.389963 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:42.389920    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:42.582473 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:42.582432    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:42.864358 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:42.864325    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:42.961437 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:42.961401    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"]
Oct 30 23:01:43.060477 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:43.060434    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:45.370007 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:45.369968    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:45.683657 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:45.683554    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:45.727982 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:45.727948    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559"]
Oct 30 23:01:45.735167 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:45.735139    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559"
Oct 30 23:01:45.737474 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:45.737443    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg"]
Oct 30 23:01:45.742978 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:45.742955    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg"
Oct 30 23:01:45.753867 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:45.753798    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559"]
Oct 30 23:01:45.776378 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:45.761103    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-bff950\""
Oct 30 23:01:45.776378 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:45.761131    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-bff950-dockercfg-9k5q4\""
Oct 30 23:01:45.776378 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:45.761559    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg"]
Oct 30 23:01:45.842742 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:45.842692    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba4dc809-cd8c-4957-ade8-9d9d434e20c9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559\" (UID: \"ba4dc809-cd8c-4957-ade8-9d9d434e20c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559"
Oct 30 23:01:45.842982 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:45.842893    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6881b170-1a5e-4c71-8fc9-c45349847afd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg\" (UID: \"6881b170-1a5e-4c71-8fc9-c45349847afd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg"
Oct 30 23:01:45.843165 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:45.843141    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns5pf\" (UniqueName: \"kubernetes.io/projected/6881b170-1a5e-4c71-8fc9-c45349847afd-kube-api-access-ns5pf\") pod \"payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg\" (UID: \"6881b170-1a5e-4c71-8fc9-c45349847afd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg"
Oct 30 23:01:45.843321 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:45.843192    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjzbg\" (UniqueName: \"kubernetes.io/projected/ba4dc809-cd8c-4957-ade8-9d9d434e20c9-kube-api-access-jjzbg\") pod \"payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559\" (UID: \"ba4dc809-cd8c-4957-ade8-9d9d434e20c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559"
Oct 30 23:01:45.858979 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:45.858933    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:45.955375 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:45.948605    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns5pf\" (UniqueName: \"kubernetes.io/projected/6881b170-1a5e-4c71-8fc9-c45349847afd-kube-api-access-ns5pf\") pod \"payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg\" (UID: \"6881b170-1a5e-4c71-8fc9-c45349847afd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg"
Oct 30 23:01:45.955375 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:45.948699    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjzbg\" (UniqueName: \"kubernetes.io/projected/ba4dc809-cd8c-4957-ade8-9d9d434e20c9-kube-api-access-jjzbg\") pod \"payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559\" (UID: \"ba4dc809-cd8c-4957-ade8-9d9d434e20c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559"
Oct 30 23:01:45.955375 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:45.948866    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba4dc809-cd8c-4957-ade8-9d9d434e20c9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559\" (UID: \"ba4dc809-cd8c-4957-ade8-9d9d434e20c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559"
Oct 30 23:01:45.955375 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:45.948963    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6881b170-1a5e-4c71-8fc9-c45349847afd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg\" (UID: \"6881b170-1a5e-4c71-8fc9-c45349847afd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg"
Oct 30 23:01:45.955375 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:45.951733    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba4dc809-cd8c-4957-ade8-9d9d434e20c9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559\" (UID: \"ba4dc809-cd8c-4957-ade8-9d9d434e20c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559"
Oct 30 23:01:45.970634 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:45.970596    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6881b170-1a5e-4c71-8fc9-c45349847afd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg\" (UID: \"6881b170-1a5e-4c71-8fc9-c45349847afd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg"
Oct 30 23:01:45.978274 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:45.976648    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns5pf\" (UniqueName: \"kubernetes.io/projected/6881b170-1a5e-4c71-8fc9-c45349847afd-kube-api-access-ns5pf\") pod \"payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg\" (UID: \"6881b170-1a5e-4c71-8fc9-c45349847afd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg"
Oct 30 23:01:45.978274 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:45.977294    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjzbg\" (UniqueName: \"kubernetes.io/projected/ba4dc809-cd8c-4957-ade8-9d9d434e20c9-kube-api-access-jjzbg\") pod \"payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559\" (UID: \"ba4dc809-cd8c-4957-ade8-9d9d434e20c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559"
Oct 30 23:01:46.058795 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:46.058751    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559"
Oct 30 23:01:46.078525 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:46.078485    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg"
Oct 30 23:01:46.374138 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:01:46.374107    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4dc809_cd8c_4957_ade8_9d9d434e20c9.slice/crio-7701946ca7b18faf4938f787270b36dc63c4fd29c61249b371a8039ba5c49fc8 WatchSource:0}: Error finding container 7701946ca7b18faf4938f787270b36dc63c4fd29c61249b371a8039ba5c49fc8: Status 404 returned error can't find the container with id 7701946ca7b18faf4938f787270b36dc63c4fd29c61249b371a8039ba5c49fc8
Oct 30 23:01:46.374736 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:46.374714    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559"]
Oct 30 23:01:46.408894 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:46.408863    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg"]
Oct 30 23:01:46.418973 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:01:46.418945    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6881b170_1a5e_4c71_8fc9_c45349847afd.slice/crio-af962a1d3c38f8cfa68e4829115cbe0c8aff779bb2f0f4b6f26272f9c512a6eb WatchSource:0}: Error finding container af962a1d3c38f8cfa68e4829115cbe0c8aff779bb2f0f4b6f26272f9c512a6eb: Status 404 returned error can't find the container with id af962a1d3c38f8cfa68e4829115cbe0c8aff779bb2f0f4b6f26272f9c512a6eb
Oct 30 23:01:47.088516 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:47.088476    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559" event={"ID":"ba4dc809-cd8c-4957-ade8-9d9d434e20c9","Type":"ContainerStarted","Data":"7a7dba4728cf1f3e0638bf2194ab4cc2d73f31fa4190bf816e9a761f17e698a8"}
Oct 30 23:01:47.088744 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:47.088550    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559"
Oct 30 23:01:47.088744 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:47.088570    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559" event={"ID":"ba4dc809-cd8c-4957-ade8-9d9d434e20c9","Type":"ContainerStarted","Data":"7701946ca7b18faf4938f787270b36dc63c4fd29c61249b371a8039ba5c49fc8"}
Oct 30 23:01:47.090430 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:47.090402    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg" event={"ID":"6881b170-1a5e-4c71-8fc9-c45349847afd","Type":"ContainerStarted","Data":"38a2e8add39d24bff838dc985e64e4caf3ab2f0b08ed7111c17663068cdc3c90"}
Oct 30 23:01:47.090565 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:47.090449    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg" event={"ID":"6881b170-1a5e-4c71-8fc9-c45349847afd","Type":"ContainerStarted","Data":"af962a1d3c38f8cfa68e4829115cbe0c8aff779bb2f0f4b6f26272f9c512a6eb"}
Oct 30 23:01:47.090565 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:47.090496    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg"
Oct 30 23:01:47.138196 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:47.138131    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559" podStartSLOduration=2.13811377 podStartE2EDuration="2.13811377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:01:47.135734634 +0000 UTC m=+3292.492100950" watchObservedRunningTime="2026-10-30 23:01:47.13811377 +0000 UTC m=+3292.494480087"
Oct 30 23:01:47.149177 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:47.149143    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:47.179835 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:47.179721    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg" podStartSLOduration=2.17970707 podStartE2EDuration="2.17970707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:01:47.176701092 +0000 UTC m=+3292.533067407" watchObservedRunningTime="2026-10-30 23:01:47.17970707 +0000 UTC m=+3292.536073384"
Oct 30 23:01:48.949900 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:48.949826    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:49.017098 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:49.017061    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf"
Oct 30 23:01:50.827937 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:50.827897    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:51.482600 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:51.482564    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:52.605345 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:52.605286    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:53.679582 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:53.679545    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s"]
Oct 30 23:01:53.845938 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:53.845898    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:54.312043 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:54.312006    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:54.511818 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:54.511779    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:55.777027 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:55.776998    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 23:01:55.783207 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:55.783178    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 23:01:55.931948 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:55.931909    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:56.121715 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:56.121624    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:56.285311 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:56.285273    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:56.496818 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:56.496750    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:56.667441 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:56.667389    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:56.832677 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:56.832581    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:57.168647 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:57.168554    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:57.992359 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:57.992312    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r" podUID="df0f924c-9c38-4e79-afee-85e96d72af9a" containerName="main" containerID="cri-o://1acf8c4182a28eae86167db482bc123cd0b9de27e3b4aa4b6e07f5faddda1233" gracePeriod=45
Oct 30 23:01:58.102566 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:58.102532    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg"
Oct 30 23:01:58.102741 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:58.102639    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559"
Oct 30 23:01:58.196352 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:58.196317    4889 generic.go:356] "Generic (PLEG): container finished" podID="df0f924c-9c38-4e79-afee-85e96d72af9a" containerID="1acf8c4182a28eae86167db482bc123cd0b9de27e3b4aa4b6e07f5faddda1233" exitCode=0
Oct 30 23:01:58.196582 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:58.196390    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r" event={"ID":"df0f924c-9c38-4e79-afee-85e96d72af9a","Type":"ContainerDied","Data":"1acf8c4182a28eae86167db482bc123cd0b9de27e3b4aa4b6e07f5faddda1233"}
Oct 30 23:01:58.338374 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:58.338346    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"
Oct 30 23:01:58.366980 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:58.366937    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-tmp-dir\") pod \"df0f924c-9c38-4e79-afee-85e96d72af9a\" (UID: \"df0f924c-9c38-4e79-afee-85e96d72af9a\") "
Oct 30 23:01:58.367375 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:58.367027    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df0f924c-9c38-4e79-afee-85e96d72af9a-kube-api-access-6mj5r\" (UniqueName: \"kubernetes.io/projected/df0f924c-9c38-4e79-afee-85e96d72af9a-kube-api-access-6mj5r\") pod \"df0f924c-9c38-4e79-afee-85e96d72af9a\" (UID: \"df0f924c-9c38-4e79-afee-85e96d72af9a\") "
Oct 30 23:01:58.367375 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:58.367070    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df0f924c-9c38-4e79-afee-85e96d72af9a-tls-certs\" (UniqueName: \"kubernetes.io/secret/df0f924c-9c38-4e79-afee-85e96d72af9a-tls-certs\") pod \"df0f924c-9c38-4e79-afee-85e96d72af9a\" (UID: \"df0f924c-9c38-4e79-afee-85e96d72af9a\") "
Oct 30 23:01:58.367375 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:58.367155    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-model-cache\") pod \"df0f924c-9c38-4e79-afee-85e96d72af9a\" (UID: \"df0f924c-9c38-4e79-afee-85e96d72af9a\") "
Oct 30 23:01:58.367375 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:58.367261    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-home\" (UniqueName: \"kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-home\") pod \"df0f924c-9c38-4e79-afee-85e96d72af9a\" (UID: \"df0f924c-9c38-4e79-afee-85e96d72af9a\") "
Oct 30 23:01:58.367375 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:58.367319    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-tmp-dir" pod "df0f924c-9c38-4e79-afee-85e96d72af9a" (UID: "df0f924c-9c38-4e79-afee-85e96d72af9a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:01:58.367375 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:58.367346    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-dshm\") pod \"df0f924c-9c38-4e79-afee-85e96d72af9a\" (UID: \"df0f924c-9c38-4e79-afee-85e96d72af9a\") "
Oct 30 23:01:58.367723 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:58.367523    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-model-cache" pod "df0f924c-9c38-4e79-afee-85e96d72af9a" (UID: "df0f924c-9c38-4e79-afee-85e96d72af9a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:01:58.367723 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:58.367553    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-home" pod "df0f924c-9c38-4e79-afee-85e96d72af9a" (UID: "df0f924c-9c38-4e79-afee-85e96d72af9a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:01:58.368039 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:58.368014    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:01:58.368039 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:58.368043    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:01:58.368297 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:58.368053    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:01:58.369655 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:58.369617    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-dshm" pod "df0f924c-9c38-4e79-afee-85e96d72af9a" (UID: "df0f924c-9c38-4e79-afee-85e96d72af9a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:01:58.369759 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:58.369706    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df0f924c-9c38-4e79-afee-85e96d72af9a-tls-certs" pod "df0f924c-9c38-4e79-afee-85e96d72af9a" (UID: "df0f924c-9c38-4e79-afee-85e96d72af9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:01:58.369806 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:58.369769    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df0f924c-9c38-4e79-afee-85e96d72af9a-kube-api-access-6mj5r" pod "df0f924c-9c38-4e79-afee-85e96d72af9a" (UID: "df0f924c-9c38-4e79-afee-85e96d72af9a"). InnerVolumeSpecName "kube-api-access-6mj5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:01:58.470017 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:58.469968    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df0f924c-9c38-4e79-afee-85e96d72af9a-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:01:58.470017 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:58.470011    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mj5r\" (UniqueName: \"kubernetes.io/projected/df0f924c-9c38-4e79-afee-85e96d72af9a-kube-api-access-6mj5r\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:01:58.470017 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:58.470024    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df0f924c-9c38-4e79-afee-85e96d72af9a-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:01:58.680407 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:58.680299    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s" podUID="c54084a2-132f-437f-a13d-42f118c87251" containerName="payload-pre-processing" containerID="cri-o://f5bf34517fb525f2b88f680a02c7940bcc5a7050060bd7654a8e39a326904b04" gracePeriod=25
Oct 30 23:01:59.206529 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:59.206493    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"
Oct 30 23:01:59.206529 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:59.206520    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r" event={"ID":"df0f924c-9c38-4e79-afee-85e96d72af9a","Type":"ContainerDied","Data":"9db82784faa3b5655294add357d0c602efd877de471af1a610e4762887d810ac"}
Oct 30 23:01:59.207169 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:59.206575    4889 scope.go:122] "RemoveContainer" containerID="1acf8c4182a28eae86167db482bc123cd0b9de27e3b4aa4b6e07f5faddda1233"
Oct 30 23:01:59.235270 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:59.235209    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:59.271384 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:59.271344    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"]
Oct 30 23:01:59.285655 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:01:59.285618    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-510f80/rate-test-model-510f80-kserve-c948fb597-bnk7r"]
Oct 30 23:02:00.777191 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:00.777151    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df0f924c-9c38-4e79-afee-85e96d72af9a" path="/var/lib/kubelet/pods/df0f924c-9c38-4e79-afee-85e96d72af9a/volumes"
Oct 30 23:02:01.105138 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:01.105011    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:03.201812 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:03.201751    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:03.389754 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:03.389696    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:03.566346 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:03.566297    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:05.271949 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:05.271909    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:07.057978 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:07.057934    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:07.963447 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:07.963408    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s" podUID="c54084a2-132f-437f-a13d-42f118c87251" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:02:09.339671 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:09.339634    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:10.306919 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:02:10.306881    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:02:11.316478 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:11.316438    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:12.715696 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:12.715654    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:14.615065 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:14.615031    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:14.619120 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:14.619083    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52"]
Oct 30 23:02:14.659700 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:14.659663    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt"]
Oct 30 23:02:14.827176 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:14.827141    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:14.998552 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:14.998516    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:16.603944 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:16.603885    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:17.926565 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:17.926529    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w"]
Oct 30 23:02:17.927838 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:17.927803    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w" podUID="eabd400f-7449-4a3b-b4ca-d969e6789f23" containerName="maas-api" containerID="cri-o://cc7edb72a5416e74269e4e05015ee87a47e7c0bac125dc3dcb2578ced15590bb" gracePeriod=30
Oct 30 23:02:17.967546 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:17.967490    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s" podUID="c54084a2-132f-437f-a13d-42f118c87251" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:02:18.041551 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:18.041520    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:18.310415 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:18.310385    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w"
Oct 30 23:02:18.381250 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:18.377389    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eabd400f-7449-4a3b-b4ca-d969e6789f23-kube-api-access-2kz2c\" (UniqueName: \"kubernetes.io/projected/eabd400f-7449-4a3b-b4ca-d969e6789f23-kube-api-access-2kz2c\") pod \"eabd400f-7449-4a3b-b4ca-d969e6789f23\" (UID: \"eabd400f-7449-4a3b-b4ca-d969e6789f23\") "
Oct 30 23:02:18.381250 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:18.377451    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eabd400f-7449-4a3b-b4ca-d969e6789f23-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabd400f-7449-4a3b-b4ca-d969e6789f23-openshift-service-ca\") pod \"eabd400f-7449-4a3b-b4ca-d969e6789f23\" (UID: \"eabd400f-7449-4a3b-b4ca-d969e6789f23\") "
Oct 30 23:02:18.381250 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:18.377503    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eabd400f-7449-4a3b-b4ca-d969e6789f23-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabd400f-7449-4a3b-b4ca-d969e6789f23-metrics-certs\") pod \"eabd400f-7449-4a3b-b4ca-d969e6789f23\" (UID: \"eabd400f-7449-4a3b-b4ca-d969e6789f23\") "
Oct 30 23:02:18.381250 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:18.377557    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eabd400f-7449-4a3b-b4ca-d969e6789f23-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eabd400f-7449-4a3b-b4ca-d969e6789f23-maas-api-tls\") pod \"eabd400f-7449-4a3b-b4ca-d969e6789f23\" (UID: \"eabd400f-7449-4a3b-b4ca-d969e6789f23\") "
Oct 30 23:02:18.393263 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:18.392096    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eabd400f-7449-4a3b-b4ca-d969e6789f23-openshift-service-ca" pod "eabd400f-7449-4a3b-b4ca-d969e6789f23" (UID: "eabd400f-7449-4a3b-b4ca-d969e6789f23"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:18.403248 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:18.399703    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eabd400f-7449-4a3b-b4ca-d969e6789f23-metrics-certs" pod "eabd400f-7449-4a3b-b4ca-d969e6789f23" (UID: "eabd400f-7449-4a3b-b4ca-d969e6789f23"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:02:18.403248 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:18.400187    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eabd400f-7449-4a3b-b4ca-d969e6789f23-kube-api-access-2kz2c" pod "eabd400f-7449-4a3b-b4ca-d969e6789f23" (UID: "eabd400f-7449-4a3b-b4ca-d969e6789f23"). InnerVolumeSpecName "kube-api-access-2kz2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:18.435360 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:18.434793    4889 generic.go:356] "Generic (PLEG): container finished" podID="eabd400f-7449-4a3b-b4ca-d969e6789f23" containerID="cc7edb72a5416e74269e4e05015ee87a47e7c0bac125dc3dcb2578ced15590bb" exitCode=0
Oct 30 23:02:18.435360 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:18.435008    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w" event={"ID":"eabd400f-7449-4a3b-b4ca-d969e6789f23","Type":"ContainerDied","Data":"cc7edb72a5416e74269e4e05015ee87a47e7c0bac125dc3dcb2578ced15590bb"}
Oct 30 23:02:18.435360 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:18.435059    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w" event={"ID":"eabd400f-7449-4a3b-b4ca-d969e6789f23","Type":"ContainerDied","Data":"c909453bfbd61863c755cb85c95a9d00b3ad6c0042a45f42e52e41ba5773c9c9"}
Oct 30 23:02:18.435360 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:18.435085    4889 scope.go:122] "RemoveContainer" containerID="cc7edb72a5416e74269e4e05015ee87a47e7c0bac125dc3dcb2578ced15590bb"
Oct 30 23:02:18.435360 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:18.435264    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w"
Oct 30 23:02:18.448506 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:18.443462    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eabd400f-7449-4a3b-b4ca-d969e6789f23-maas-api-tls" pod "eabd400f-7449-4a3b-b4ca-d969e6789f23" (UID: "eabd400f-7449-4a3b-b4ca-d969e6789f23"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:02:18.486068 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:18.485914    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kz2c\" (UniqueName: \"kubernetes.io/projected/eabd400f-7449-4a3b-b4ca-d969e6789f23-kube-api-access-2kz2c\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:18.486068 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:18.485943    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabd400f-7449-4a3b-b4ca-d969e6789f23-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:18.486068 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:18.485954    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabd400f-7449-4a3b-b4ca-d969e6789f23-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:18.486068 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:18.485964    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eabd400f-7449-4a3b-b4ca-d969e6789f23-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:18.489152 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:18.489131    4889 scope.go:122] "RemoveContainer" containerID="cc7edb72a5416e74269e4e05015ee87a47e7c0bac125dc3dcb2578ced15590bb"
Oct 30 23:02:18.489535 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:02:18.489509    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc7edb72a5416e74269e4e05015ee87a47e7c0bac125dc3dcb2578ced15590bb\": container with ID starting with cc7edb72a5416e74269e4e05015ee87a47e7c0bac125dc3dcb2578ced15590bb not found: ID does not exist" containerID="cc7edb72a5416e74269e4e05015ee87a47e7c0bac125dc3dcb2578ced15590bb"
Oct 30 23:02:18.489599 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:18.489545    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc7edb72a5416e74269e4e05015ee87a47e7c0bac125dc3dcb2578ced15590bb"} err="failed to get container status \"cc7edb72a5416e74269e4e05015ee87a47e7c0bac125dc3dcb2578ced15590bb\": rpc error: code = NotFound desc = could not find container \"cc7edb72a5416e74269e4e05015ee87a47e7c0bac125dc3dcb2578ced15590bb\": container with ID starting with cc7edb72a5416e74269e4e05015ee87a47e7c0bac125dc3dcb2578ced15590bb not found: ID does not exist"
Oct 30 23:02:18.668486 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:18.668403    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:18.794813 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:18.794775    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w"]
Oct 30 23:02:18.802456 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:18.802418    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0ff5b0-557cd59c6c-cwj2w"]
Oct 30 23:02:18.841252 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:18.839937    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:18.969022 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:18.968996    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s"
Oct 30 23:02:19.094051 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:19.094013    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c54084a2-132f-437f-a13d-42f118c87251-kube-api-access-jz9qm\" (UniqueName: \"kubernetes.io/projected/c54084a2-132f-437f-a13d-42f118c87251-kube-api-access-jz9qm\") pod \"c54084a2-132f-437f-a13d-42f118c87251\" (UID: \"c54084a2-132f-437f-a13d-42f118c87251\") "
Oct 30 23:02:19.094288 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:19.094077    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c54084a2-132f-437f-a13d-42f118c87251-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c54084a2-132f-437f-a13d-42f118c87251-plugins-config-volume\") pod \"c54084a2-132f-437f-a13d-42f118c87251\" (UID: \"c54084a2-132f-437f-a13d-42f118c87251\") "
Oct 30 23:02:19.094474 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:19.094451    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c54084a2-132f-437f-a13d-42f118c87251-plugins-config-volume" pod "c54084a2-132f-437f-a13d-42f118c87251" (UID: "c54084a2-132f-437f-a13d-42f118c87251"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:19.094825 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:19.094804    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c54084a2-132f-437f-a13d-42f118c87251-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:19.096477 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:19.096457    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c54084a2-132f-437f-a13d-42f118c87251-kube-api-access-jz9qm" pod "c54084a2-132f-437f-a13d-42f118c87251" (UID: "c54084a2-132f-437f-a13d-42f118c87251"). InnerVolumeSpecName "kube-api-access-jz9qm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:19.196690 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:19.196650    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz9qm\" (UniqueName: \"kubernetes.io/projected/c54084a2-132f-437f-a13d-42f118c87251-kube-api-access-jz9qm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:19.281601 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:19.281564    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:19.445026 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:19.444985    4889 generic.go:356] "Generic (PLEG): container finished" podID="c54084a2-132f-437f-a13d-42f118c87251" containerID="f5bf34517fb525f2b88f680a02c7940bcc5a7050060bd7654a8e39a326904b04" exitCode=0
Oct 30 23:02:19.445263 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:19.445055    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s"
Oct 30 23:02:19.445263 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:19.445068    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s" event={"ID":"c54084a2-132f-437f-a13d-42f118c87251","Type":"ContainerDied","Data":"f5bf34517fb525f2b88f680a02c7940bcc5a7050060bd7654a8e39a326904b04"}
Oct 30 23:02:19.445263 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:19.445128    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s" event={"ID":"c54084a2-132f-437f-a13d-42f118c87251","Type":"ContainerDied","Data":"60574b61ad7fc0e4d483931228b2e8193dd23fbfdf462e8247ab348968e0ce64"}
Oct 30 23:02:19.445263 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:19.445157    4889 scope.go:122] "RemoveContainer" containerID="f5bf34517fb525f2b88f680a02c7940bcc5a7050060bd7654a8e39a326904b04"
Oct 30 23:02:19.456962 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:19.456938    4889 scope.go:122] "RemoveContainer" containerID="f5bf34517fb525f2b88f680a02c7940bcc5a7050060bd7654a8e39a326904b04"
Oct 30 23:02:19.457285 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:02:19.457265    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5bf34517fb525f2b88f680a02c7940bcc5a7050060bd7654a8e39a326904b04\": container with ID starting with f5bf34517fb525f2b88f680a02c7940bcc5a7050060bd7654a8e39a326904b04 not found: ID does not exist" containerID="f5bf34517fb525f2b88f680a02c7940bcc5a7050060bd7654a8e39a326904b04"
Oct 30 23:02:19.457374 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:19.457293    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5bf34517fb525f2b88f680a02c7940bcc5a7050060bd7654a8e39a326904b04"} err="failed to get container status \"f5bf34517fb525f2b88f680a02c7940bcc5a7050060bd7654a8e39a326904b04\": rpc error: code = NotFound desc = could not find container \"f5bf34517fb525f2b88f680a02c7940bcc5a7050060bd7654a8e39a326904b04\": container with ID starting with f5bf34517fb525f2b88f680a02c7940bcc5a7050060bd7654a8e39a326904b04 not found: ID does not exist"
Oct 30 23:02:19.503331 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:19.503292    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s"]
Oct 30 23:02:19.518860 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:19.518816    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6f2eba-dc7c6b979-rb48s"]
Oct 30 23:02:19.619813 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:19.619717    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52" podUID="9683180a-1f65-48bb-a62c-c78cef5799a3" containerName="payload-processing" containerID="cri-o://3250ae08a08187eccf3a2f33c7b06aeb17c5b15ced73b0b490efe86572b354ea" gracePeriod=25
Oct 30 23:02:19.660562 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:19.660519    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt" podUID="63b2f029-b001-48d1-8c60-237096ebe16e" containerName="payload-pre-processing" containerID="cri-o://2418049e857e65cd08a191d3598901910c138abf046ca4c7101712edd89d5f03" gracePeriod=25
Oct 30 23:02:20.576006 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:20.575967    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7269-09466f"]
Oct 30 23:02:20.576795 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:20.576777    4889 cpu_manager.go:414] "containerMap: removing container" podUID="eabd400f-7449-4a3b-b4ca-d969e6789f23" containerName="maas-api"
Oct 30 23:02:20.576844 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:20.576796    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eabd400f-7449-4a3b-b4ca-d969e6789f23" containerName="maas-api"
Oct 30 23:02:20.576844 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:20.576819    4889 cpu_manager.go:414] "containerMap: removing container" podUID="c54084a2-132f-437f-a13d-42f118c87251" containerName="payload-pre-processing"
Oct 30 23:02:20.576844 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:20.576824    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c54084a2-132f-437f-a13d-42f118c87251" containerName="payload-pre-processing"
Oct 30 23:02:20.576962 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:20.576847    4889 cpu_manager.go:414] "containerMap: removing container" podUID="df0f924c-9c38-4e79-afee-85e96d72af9a" containerName="main"
Oct 30 23:02:20.576962 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:20.576852    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df0f924c-9c38-4e79-afee-85e96d72af9a" containerName="main"
Oct 30 23:02:20.577033 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:20.576962    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="c54084a2-132f-437f-a13d-42f118c87251" containerName="payload-pre-processing"
Oct 30 23:02:20.577033 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:20.576974    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="df0f924c-9c38-4e79-afee-85e96d72af9a" containerName="main"
Oct 30 23:02:20.577033 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:20.576987    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="eabd400f-7449-4a3b-b4ca-d969e6789f23" containerName="maas-api"
Oct 30 23:02:20.582174 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:20.582156    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-09466f"
Oct 30 23:02:20.594908 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:20.594876    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7269-09466f"]
Oct 30 23:02:20.625414 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:20.625382    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xbw52\""
Oct 30 23:02:20.723933 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:20.723886    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l4n6\" (UniqueName: \"kubernetes.io/projected/15e7f976-2726-4949-b6f7-31d4a362a5d4-kube-api-access-4l4n6\") pod \"test-curl-7269-09466f\" (UID: \"15e7f976-2726-4949-b6f7-31d4a362a5d4\") " pod="opendatahub/test-curl-7269-09466f"
Oct 30 23:02:20.726546 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:20.726515    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:20.778522 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:20.778490    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c54084a2-132f-437f-a13d-42f118c87251" path="/var/lib/kubelet/pods/c54084a2-132f-437f-a13d-42f118c87251/volumes"
Oct 30 23:02:20.778999 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:20.778983    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eabd400f-7449-4a3b-b4ca-d969e6789f23" path="/var/lib/kubelet/pods/eabd400f-7449-4a3b-b4ca-d969e6789f23/volumes"
Oct 30 23:02:20.826133 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:20.826050    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l4n6\" (UniqueName: \"kubernetes.io/projected/15e7f976-2726-4949-b6f7-31d4a362a5d4-kube-api-access-4l4n6\") pod \"test-curl-7269-09466f\" (UID: \"15e7f976-2726-4949-b6f7-31d4a362a5d4\") " pod="opendatahub/test-curl-7269-09466f"
Oct 30 23:02:20.847351 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:20.847317    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l4n6\" (UniqueName: \"kubernetes.io/projected/15e7f976-2726-4949-b6f7-31d4a362a5d4-kube-api-access-4l4n6\") pod \"test-curl-7269-09466f\" (UID: \"15e7f976-2726-4949-b6f7-31d4a362a5d4\") " pod="opendatahub/test-curl-7269-09466f"
Oct 30 23:02:20.898547 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:20.898512    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-09466f"
Oct 30 23:02:21.219437 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:02:21.219403    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15e7f976_2726_4949_b6f7_31d4a362a5d4.slice/crio-6d27b82edabcf1621b83e6ad336564f6be501c19a3c36c26ecc9b5b521d43923 WatchSource:0}: Error finding container 6d27b82edabcf1621b83e6ad336564f6be501c19a3c36c26ecc9b5b521d43923: Status 404 returned error can't find the container with id 6d27b82edabcf1621b83e6ad336564f6be501c19a3c36c26ecc9b5b521d43923
Oct 30 23:02:21.220171 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:21.220137    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7269-09466f"]
Oct 30 23:02:21.468526 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:21.468423    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7269-09466f" event={"ID":"15e7f976-2726-4949-b6f7-31d4a362a5d4","Type":"ContainerStarted","Data":"d9501fb8dd9fc329a5fad05b2984346c44e1d25d65aade207835adafb701744a"}
Oct 30 23:02:21.468526 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:21.468478    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7269-09466f" event={"ID":"15e7f976-2726-4949-b6f7-31d4a362a5d4","Type":"ContainerStarted","Data":"6d27b82edabcf1621b83e6ad336564f6be501c19a3c36c26ecc9b5b521d43923"}
Oct 30 23:02:21.509493 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:21.509433    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7269-09466f" podStartSLOduration=1.5094198300000001 podStartE2EDuration="1.50941983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:02:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:02:21.508286083 +0000 UTC m=+3326.864652395" watchObservedRunningTime="2026-10-30 23:02:21.50941983 +0000 UTC m=+3326.865786146"
Oct 30 23:02:22.381512 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:22.381465    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7269-09466f"]
Oct 30 23:02:22.390591 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:22.390557    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7269-09466f"]
Oct 30 23:02:23.112160 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:23.112117    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:23.288448 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:23.288410    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:23.484244 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:23.484104    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7269-09466f" podUID="15e7f976-2726-4949-b6f7-31d4a362a5d4" containerName="test-curl-7269-09466f" containerID="cri-o://d9501fb8dd9fc329a5fad05b2984346c44e1d25d65aade207835adafb701744a" gracePeriod=2
Oct 30 23:02:24.816764 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:24.816718    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:25.395338 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:25.395302    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:25.570388 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:25.569278    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:25.593093 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:25.588673    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"]
Oct 30 23:02:25.593093 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:25.588957    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h" podUID="7cfa769f-0310-4749-a929-646c9ab9bea5" containerName="istio-proxy" containerID="cri-o://64c2ef9d16cd8cf15f1e6a5ef4ac7ebfe4753234b9a225c85047034b91bf4682" gracePeriod=30
Oct 30 23:02:25.771250 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:25.770880    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:25.848610 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:25.848579    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-09466f"
Oct 30 23:02:25.915582 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:25.915486    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15e7f976-2726-4949-b6f7-31d4a362a5d4-kube-api-access-4l4n6\" (UniqueName: \"kubernetes.io/projected/15e7f976-2726-4949-b6f7-31d4a362a5d4-kube-api-access-4l4n6\") pod \"15e7f976-2726-4949-b6f7-31d4a362a5d4\" (UID: \"15e7f976-2726-4949-b6f7-31d4a362a5d4\") "
Oct 30 23:02:25.917806 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:25.917781    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e7f976-2726-4949-b6f7-31d4a362a5d4-kube-api-access-4l4n6" pod "15e7f976-2726-4949-b6f7-31d4a362a5d4" (UID: "15e7f976-2726-4949-b6f7-31d4a362a5d4"). InnerVolumeSpecName "kube-api-access-4l4n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:26.017150 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:26.017115    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l4n6\" (UniqueName: \"kubernetes.io/projected/15e7f976-2726-4949-b6f7-31d4a362a5d4-kube-api-access-4l4n6\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:26.522315 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:26.522277    4889 generic.go:356] "Generic (PLEG): container finished" podID="15e7f976-2726-4949-b6f7-31d4a362a5d4" containerID="d9501fb8dd9fc329a5fad05b2984346c44e1d25d65aade207835adafb701744a" exitCode=137
Oct 30 23:02:26.522525 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:26.522352    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-09466f"
Oct 30 23:02:26.522525 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:26.522379    4889 scope.go:122] "RemoveContainer" containerID="d9501fb8dd9fc329a5fad05b2984346c44e1d25d65aade207835adafb701744a"
Oct 30 23:02:26.535753 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:26.535730    4889 scope.go:122] "RemoveContainer" containerID="d9501fb8dd9fc329a5fad05b2984346c44e1d25d65aade207835adafb701744a"
Oct 30 23:02:26.536122 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:02:26.536100    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9501fb8dd9fc329a5fad05b2984346c44e1d25d65aade207835adafb701744a\": container with ID starting with d9501fb8dd9fc329a5fad05b2984346c44e1d25d65aade207835adafb701744a not found: ID does not exist" containerID="d9501fb8dd9fc329a5fad05b2984346c44e1d25d65aade207835adafb701744a"
Oct 30 23:02:26.536173 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:26.536133    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9501fb8dd9fc329a5fad05b2984346c44e1d25d65aade207835adafb701744a"} err="failed to get container status \"d9501fb8dd9fc329a5fad05b2984346c44e1d25d65aade207835adafb701744a\": rpc error: code = NotFound desc = could not find container \"d9501fb8dd9fc329a5fad05b2984346c44e1d25d65aade207835adafb701744a\": container with ID starting with d9501fb8dd9fc329a5fad05b2984346c44e1d25d65aade207835adafb701744a not found: ID does not exist"
Oct 30 23:02:26.547878 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:02:26.547838    4889 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7269-09466f\" is forbidden: User \"system:node:ip-10-0-142-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-185.ec2.internal' and this object" podUID="15e7f976-2726-4949-b6f7-31d4a362a5d4" pod="opendatahub/test-curl-7269-09466f"
Oct 30 23:02:26.651956 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:26.651915    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:26.777837 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:26.777759    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15e7f976-2726-4949-b6f7-31d4a362a5d4" path="/var/lib/kubelet/pods/15e7f976-2726-4949-b6f7-31d4a362a5d4/volumes"
Oct 30 23:02:26.814501 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:26.814463    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt" podUID="63b2f029-b001-48d1-8c60-237096ebe16e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:02:26.814687 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:26.814469    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52" podUID="9683180a-1f65-48bb-a62c-c78cef5799a3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:02:28.397602 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:28.397559    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:28.565968 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:28.565914    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:29.907644 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:29.907611    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf"]
Oct 30 23:02:29.908363 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:29.907829    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf" podUID="1c1dcc08-1daf-4112-93d9-036d7dd70f6c" containerName="maas-api" containerID="cri-o://2f7275f3044311f78eab8b80e48ddf6507a50b3ceef5ff97590a1d7c86c006fe" gracePeriod=30
Oct 30 23:02:30.043121 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.043085    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:30.082467 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.082420    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m"]
Oct 30 23:02:30.082654 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.082607    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m" podUID="8cf81220-6bf3-42ab-8330-ed9a73dc8190" containerName="payload-processing" containerID="cri-o://684e20da2d668adb6f55655eb527fdac9a36412932867ef09a81cfdf3d6baf43" gracePeriod=30
Oct 30 23:02:30.107208 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.107160    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n"]
Oct 30 23:02:30.107784 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.107543    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n" podUID="3d913d76-fad1-4092-bab7-e13c74cd2e0f" containerName="payload-pre-processing" containerID="cri-o://c31e2367c7a7cf2738859b3d7ab70c05d6dc49704f5b99bdc255d84d9efb78ca" gracePeriod=30
Oct 30 23:02:30.222161 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.222039    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:30.259922 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.259452    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf"
Oct 30 23:02:30.281069 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.281035    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-maas-api-tls\") pod \"1c1dcc08-1daf-4112-93d9-036d7dd70f6c\" (UID: \"1c1dcc08-1daf-4112-93d9-036d7dd70f6c\") "
Oct 30 23:02:30.281316 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.281144    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-metrics-certs\") pod \"1c1dcc08-1daf-4112-93d9-036d7dd70f6c\" (UID: \"1c1dcc08-1daf-4112-93d9-036d7dd70f6c\") "
Oct 30 23:02:30.281402 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.281352    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-openshift-service-ca\") pod \"1c1dcc08-1daf-4112-93d9-036d7dd70f6c\" (UID: \"1c1dcc08-1daf-4112-93d9-036d7dd70f6c\") "
Oct 30 23:02:30.281462 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.281411    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-kube-api-access-8gqr6\" (UniqueName: \"kubernetes.io/projected/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-kube-api-access-8gqr6\") pod \"1c1dcc08-1daf-4112-93d9-036d7dd70f6c\" (UID: \"1c1dcc08-1daf-4112-93d9-036d7dd70f6c\") "
Oct 30 23:02:30.282018 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.281987    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-openshift-service-ca" pod "1c1dcc08-1daf-4112-93d9-036d7dd70f6c" (UID: "1c1dcc08-1daf-4112-93d9-036d7dd70f6c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:30.286169 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.286137    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-maas-api-tls" pod "1c1dcc08-1daf-4112-93d9-036d7dd70f6c" (UID: "1c1dcc08-1daf-4112-93d9-036d7dd70f6c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:02:30.286321 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.286252    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-metrics-certs" pod "1c1dcc08-1daf-4112-93d9-036d7dd70f6c" (UID: "1c1dcc08-1daf-4112-93d9-036d7dd70f6c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:02:30.286321 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.286292    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-kube-api-access-8gqr6" pod "1c1dcc08-1daf-4112-93d9-036d7dd70f6c" (UID: "1c1dcc08-1daf-4112-93d9-036d7dd70f6c"). InnerVolumeSpecName "kube-api-access-8gqr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:30.385281 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.385240    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:30.385281 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.385282    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gqr6\" (UniqueName: \"kubernetes.io/projected/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-kube-api-access-8gqr6\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:30.385525 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.385298    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:30.385525 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.385312    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c1dcc08-1daf-4112-93d9-036d7dd70f6c-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:30.399696 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.399597    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:30.404129 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.404106    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m"
Oct 30 23:02:30.506065 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.501631    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cf81220-6bf3-42ab-8330-ed9a73dc8190-kube-api-access-5z6sf\" (UniqueName: \"kubernetes.io/projected/8cf81220-6bf3-42ab-8330-ed9a73dc8190-kube-api-access-5z6sf\") pod \"8cf81220-6bf3-42ab-8330-ed9a73dc8190\" (UID: \"8cf81220-6bf3-42ab-8330-ed9a73dc8190\") "
Oct 30 23:02:30.506065 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.501754    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cf81220-6bf3-42ab-8330-ed9a73dc8190-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cf81220-6bf3-42ab-8330-ed9a73dc8190-plugins-config-volume\") pod \"8cf81220-6bf3-42ab-8330-ed9a73dc8190\" (UID: \"8cf81220-6bf3-42ab-8330-ed9a73dc8190\") "
Oct 30 23:02:30.531249 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.515476    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cf81220-6bf3-42ab-8330-ed9a73dc8190-plugins-config-volume" pod "8cf81220-6bf3-42ab-8330-ed9a73dc8190" (UID: "8cf81220-6bf3-42ab-8330-ed9a73dc8190"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:30.531249 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.520779    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf81220-6bf3-42ab-8330-ed9a73dc8190-kube-api-access-5z6sf" pod "8cf81220-6bf3-42ab-8330-ed9a73dc8190" (UID: "8cf81220-6bf3-42ab-8330-ed9a73dc8190"). InnerVolumeSpecName "kube-api-access-5z6sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:30.565615 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.565459    4889 generic.go:356] "Generic (PLEG): container finished" podID="3d913d76-fad1-4092-bab7-e13c74cd2e0f" containerID="c31e2367c7a7cf2738859b3d7ab70c05d6dc49704f5b99bdc255d84d9efb78ca" exitCode=0
Oct 30 23:02:30.565615 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.565543    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n" event={"ID":"3d913d76-fad1-4092-bab7-e13c74cd2e0f","Type":"ContainerDied","Data":"c31e2367c7a7cf2738859b3d7ab70c05d6dc49704f5b99bdc255d84d9efb78ca"}
Oct 30 23:02:30.570300 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.567719    4889 generic.go:356] "Generic (PLEG): container finished" podID="8cf81220-6bf3-42ab-8330-ed9a73dc8190" containerID="684e20da2d668adb6f55655eb527fdac9a36412932867ef09a81cfdf3d6baf43" exitCode=0
Oct 30 23:02:30.570300 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.567819    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m"
Oct 30 23:02:30.570300 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.567841    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m" event={"ID":"8cf81220-6bf3-42ab-8330-ed9a73dc8190","Type":"ContainerDied","Data":"684e20da2d668adb6f55655eb527fdac9a36412932867ef09a81cfdf3d6baf43"}
Oct 30 23:02:30.570300 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.567918    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m" event={"ID":"8cf81220-6bf3-42ab-8330-ed9a73dc8190","Type":"ContainerDied","Data":"ad4a0e4e8abd738c9deb7632c43efb39b1190415c4b084493bc6bf7ea49db92b"}
Oct 30 23:02:30.570300 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.567952    4889 scope.go:122] "RemoveContainer" containerID="684e20da2d668adb6f55655eb527fdac9a36412932867ef09a81cfdf3d6baf43"
Oct 30 23:02:30.570300 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.569734    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n"
Oct 30 23:02:30.575909 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.575875    4889 generic.go:356] "Generic (PLEG): container finished" podID="1c1dcc08-1daf-4112-93d9-036d7dd70f6c" containerID="2f7275f3044311f78eab8b80e48ddf6507a50b3ceef5ff97590a1d7c86c006fe" exitCode=0
Oct 30 23:02:30.576066 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.575989    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf" event={"ID":"1c1dcc08-1daf-4112-93d9-036d7dd70f6c","Type":"ContainerDied","Data":"2f7275f3044311f78eab8b80e48ddf6507a50b3ceef5ff97590a1d7c86c006fe"}
Oct 30 23:02:30.576066 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.576037    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf" event={"ID":"1c1dcc08-1daf-4112-93d9-036d7dd70f6c","Type":"ContainerDied","Data":"0e3e22c841fa2252703f2183d96c71922f58743e6632ec1606ffceffa034fe2d"}
Oct 30 23:02:30.576193 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.576110    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf"
Oct 30 23:02:30.584137 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.584111    4889 scope.go:122] "RemoveContainer" containerID="684e20da2d668adb6f55655eb527fdac9a36412932867ef09a81cfdf3d6baf43"
Oct 30 23:02:30.585633 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:02:30.584672    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"684e20da2d668adb6f55655eb527fdac9a36412932867ef09a81cfdf3d6baf43\": container with ID starting with 684e20da2d668adb6f55655eb527fdac9a36412932867ef09a81cfdf3d6baf43 not found: ID does not exist" containerID="684e20da2d668adb6f55655eb527fdac9a36412932867ef09a81cfdf3d6baf43"
Oct 30 23:02:30.585633 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.584705    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684e20da2d668adb6f55655eb527fdac9a36412932867ef09a81cfdf3d6baf43"} err="failed to get container status \"684e20da2d668adb6f55655eb527fdac9a36412932867ef09a81cfdf3d6baf43\": rpc error: code = NotFound desc = could not find container \"684e20da2d668adb6f55655eb527fdac9a36412932867ef09a81cfdf3d6baf43\": container with ID starting with 684e20da2d668adb6f55655eb527fdac9a36412932867ef09a81cfdf3d6baf43 not found: ID does not exist"
Oct 30 23:02:30.585633 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.584761    4889 scope.go:122] "RemoveContainer" containerID="2f7275f3044311f78eab8b80e48ddf6507a50b3ceef5ff97590a1d7c86c006fe"
Oct 30 23:02:30.597374 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.597352    4889 scope.go:122] "RemoveContainer" containerID="2f7275f3044311f78eab8b80e48ddf6507a50b3ceef5ff97590a1d7c86c006fe"
Oct 30 23:02:30.597728 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:02:30.597666    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f7275f3044311f78eab8b80e48ddf6507a50b3ceef5ff97590a1d7c86c006fe\": container with ID starting with 2f7275f3044311f78eab8b80e48ddf6507a50b3ceef5ff97590a1d7c86c006fe not found: ID does not exist" containerID="2f7275f3044311f78eab8b80e48ddf6507a50b3ceef5ff97590a1d7c86c006fe"
Oct 30 23:02:30.597817 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.597741    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f7275f3044311f78eab8b80e48ddf6507a50b3ceef5ff97590a1d7c86c006fe"} err="failed to get container status \"2f7275f3044311f78eab8b80e48ddf6507a50b3ceef5ff97590a1d7c86c006fe\": rpc error: code = NotFound desc = could not find container \"2f7275f3044311f78eab8b80e48ddf6507a50b3ceef5ff97590a1d7c86c006fe\": container with ID starting with 2f7275f3044311f78eab8b80e48ddf6507a50b3ceef5ff97590a1d7c86c006fe not found: ID does not exist"
Oct 30 23:02:30.603297 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.603271    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d913d76-fad1-4092-bab7-e13c74cd2e0f-kube-api-access-tw84q\" (UniqueName: \"kubernetes.io/projected/3d913d76-fad1-4092-bab7-e13c74cd2e0f-kube-api-access-tw84q\") pod \"3d913d76-fad1-4092-bab7-e13c74cd2e0f\" (UID: \"3d913d76-fad1-4092-bab7-e13c74cd2e0f\") "
Oct 30 23:02:30.603399 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.603369    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d913d76-fad1-4092-bab7-e13c74cd2e0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d913d76-fad1-4092-bab7-e13c74cd2e0f-plugins-config-volume\") pod \"3d913d76-fad1-4092-bab7-e13c74cd2e0f\" (UID: \"3d913d76-fad1-4092-bab7-e13c74cd2e0f\") "
Oct 30 23:02:30.603739 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.603707    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d913d76-fad1-4092-bab7-e13c74cd2e0f-plugins-config-volume" pod "3d913d76-fad1-4092-bab7-e13c74cd2e0f" (UID: "3d913d76-fad1-4092-bab7-e13c74cd2e0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:30.603840 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.603818    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d913d76-fad1-4092-bab7-e13c74cd2e0f-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:30.603914 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.603839    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z6sf\" (UniqueName: \"kubernetes.io/projected/8cf81220-6bf3-42ab-8330-ed9a73dc8190-kube-api-access-5z6sf\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:30.603914 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.603865    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cf81220-6bf3-42ab-8330-ed9a73dc8190-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:30.605745 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.605723    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d913d76-fad1-4092-bab7-e13c74cd2e0f-kube-api-access-tw84q" pod "3d913d76-fad1-4092-bab7-e13c74cd2e0f" (UID: "3d913d76-fad1-4092-bab7-e13c74cd2e0f"). InnerVolumeSpecName "kube-api-access-tw84q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:30.613725 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.613703    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:30.696762 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.696725    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m"]
Oct 30 23:02:30.700686 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.700656    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0a807-69fbbcbd9d-9r72m"]
Oct 30 23:02:30.705782 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.705754    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw84q\" (UniqueName: \"kubernetes.io/projected/3d913d76-fad1-4092-bab7-e13c74cd2e0f-kube-api-access-tw84q\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:30.746272 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.746137    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf"]
Oct 30 23:02:30.752175 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.752138    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0a807-b9ccf5798-v75gf"]
Oct 30 23:02:30.780282 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.780175    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c1dcc08-1daf-4112-93d9-036d7dd70f6c" path="/var/lib/kubelet/pods/1c1dcc08-1daf-4112-93d9-036d7dd70f6c/volumes"
Oct 30 23:02:30.780463 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.780416    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:30.780780 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.780758    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cf81220-6bf3-42ab-8330-ed9a73dc8190" path="/var/lib/kubelet/pods/8cf81220-6bf3-42ab-8330-ed9a73dc8190/volumes"
Oct 30 23:02:30.868384 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.868359    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 23:02:30.909437 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.909408    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-token\" (UniqueName: \"kubernetes.io/projected/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-token\") pod \"7cfa769f-0310-4749-a929-646c9ab9bea5\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") "
Oct 30 23:02:30.909877 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.909474    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cfa769f-0310-4749-a929-646c9ab9bea5-kube-api-access-6wcdl\" (UniqueName: \"kubernetes.io/projected/7cfa769f-0310-4749-a929-646c9ab9bea5-kube-api-access-6wcdl\") pod \"7cfa769f-0310-4749-a929-646c9ab9bea5\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") "
Oct 30 23:02:30.909877 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.909570    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-envoy\") pod \"7cfa769f-0310-4749-a929-646c9ab9bea5\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") "
Oct 30 23:02:30.909877 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.909609    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-podinfo\") pod \"7cfa769f-0310-4749-a929-646c9ab9bea5\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") "
Oct 30 23:02:30.909877 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.909640    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-credential-socket\") pod \"7cfa769f-0310-4749-a929-646c9ab9bea5\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") "
Oct 30 23:02:30.909877 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.909672    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-workload-certs\") pod \"7cfa769f-0310-4749-a929-646c9ab9bea5\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") "
Oct 30 23:02:30.909877 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.909741    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-data\") pod \"7cfa769f-0310-4749-a929-646c9ab9bea5\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") "
Oct 30 23:02:30.909877 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.909831    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-workload-socket\") pod \"7cfa769f-0310-4749-a929-646c9ab9bea5\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") "
Oct 30 23:02:30.910360 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.909893    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cfa769f-0310-4749-a929-646c9ab9bea5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cfa769f-0310-4749-a929-646c9ab9bea5-istiod-ca-cert\") pod \"7cfa769f-0310-4749-a929-646c9ab9bea5\" (UID: \"7cfa769f-0310-4749-a929-646c9ab9bea5\") "
Oct 30 23:02:30.910360 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.909949    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-credential-socket" pod "7cfa769f-0310-4749-a929-646c9ab9bea5" (UID: "7cfa769f-0310-4749-a929-646c9ab9bea5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:30.910360 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.909970    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-workload-certs" pod "7cfa769f-0310-4749-a929-646c9ab9bea5" (UID: "7cfa769f-0310-4749-a929-646c9ab9bea5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:30.910360 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.910059    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-data" pod "7cfa769f-0310-4749-a929-646c9ab9bea5" (UID: "7cfa769f-0310-4749-a929-646c9ab9bea5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:30.910360 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.910115    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-workload-socket" pod "7cfa769f-0310-4749-a929-646c9ab9bea5" (UID: "7cfa769f-0310-4749-a929-646c9ab9bea5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:30.910872 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.910611    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cfa769f-0310-4749-a929-646c9ab9bea5-istiod-ca-cert" pod "7cfa769f-0310-4749-a929-646c9ab9bea5" (UID: "7cfa769f-0310-4749-a929-646c9ab9bea5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:30.912355 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.912327    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:30.912497 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.912361    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cfa769f-0310-4749-a929-646c9ab9bea5-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:30.912497 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.912376    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:30.912497 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.912391    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:30.912497 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.912403    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:30.912497 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.912428    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-token" pod "7cfa769f-0310-4749-a929-646c9ab9bea5" (UID: "7cfa769f-0310-4749-a929-646c9ab9bea5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:30.912857 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.912834    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cfa769f-0310-4749-a929-646c9ab9bea5-kube-api-access-6wcdl" pod "7cfa769f-0310-4749-a929-646c9ab9bea5" (UID: "7cfa769f-0310-4749-a929-646c9ab9bea5"). InnerVolumeSpecName "kube-api-access-6wcdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:30.913089 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.913064    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-podinfo" pod "7cfa769f-0310-4749-a929-646c9ab9bea5" (UID: "7cfa769f-0310-4749-a929-646c9ab9bea5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:02:30.913208 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:30.913151    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-envoy" pod "7cfa769f-0310-4749-a929-646c9ab9bea5" (UID: "7cfa769f-0310-4749-a929-646c9ab9bea5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:31.013955 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:31.013920    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:31.013955 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:31.013954    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wcdl\" (UniqueName: \"kubernetes.io/projected/7cfa769f-0310-4749-a929-646c9ab9bea5-kube-api-access-6wcdl\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:31.014190 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:31.013968    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:31.014190 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:31.013981    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cfa769f-0310-4749-a929-646c9ab9bea5-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:31.233206 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:31.233114    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:31.587820 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:31.587788    4889 generic.go:356] "Generic (PLEG): container finished" podID="7cfa769f-0310-4749-a929-646c9ab9bea5" containerID="64c2ef9d16cd8cf15f1e6a5ef4ac7ebfe4753234b9a225c85047034b91bf4682" exitCode=0
Oct 30 23:02:31.588034 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:31.587842    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"
Oct 30 23:02:31.588034 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:31.587869    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h" event={"ID":"7cfa769f-0310-4749-a929-646c9ab9bea5","Type":"ContainerDied","Data":"64c2ef9d16cd8cf15f1e6a5ef4ac7ebfe4753234b9a225c85047034b91bf4682"}
Oct 30 23:02:31.588034 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:31.587928    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h" event={"ID":"7cfa769f-0310-4749-a929-646c9ab9bea5","Type":"ContainerDied","Data":"483e155a808ab15edf6a8cc6b59ea6156cbc40645c6a07b9567b8de424d882f2"}
Oct 30 23:02:31.588034 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:31.587952    4889 scope.go:122] "RemoveContainer" containerID="64c2ef9d16cd8cf15f1e6a5ef4ac7ebfe4753234b9a225c85047034b91bf4682"
Oct 30 23:02:31.595532 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:31.595504    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n" event={"ID":"3d913d76-fad1-4092-bab7-e13c74cd2e0f","Type":"ContainerDied","Data":"2f5975e2a02fe503244e3700f0dae1894d1f2602a1461cbcc2b10216e604ae9d"}
Oct 30 23:02:31.595678 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:31.595592    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n"
Oct 30 23:02:31.603033 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:31.603014    4889 scope.go:122] "RemoveContainer" containerID="fcd12a6635b21bdca58d709888fee4cd88f8890fa114044e2004c2c4ef80c411"
Oct 30 23:02:31.613678 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:31.613661    4889 scope.go:122] "RemoveContainer" containerID="64c2ef9d16cd8cf15f1e6a5ef4ac7ebfe4753234b9a225c85047034b91bf4682"
Oct 30 23:02:31.613942 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:02:31.613925    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64c2ef9d16cd8cf15f1e6a5ef4ac7ebfe4753234b9a225c85047034b91bf4682\": container with ID starting with 64c2ef9d16cd8cf15f1e6a5ef4ac7ebfe4753234b9a225c85047034b91bf4682 not found: ID does not exist" containerID="64c2ef9d16cd8cf15f1e6a5ef4ac7ebfe4753234b9a225c85047034b91bf4682"
Oct 30 23:02:31.613993 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:31.613948    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64c2ef9d16cd8cf15f1e6a5ef4ac7ebfe4753234b9a225c85047034b91bf4682"} err="failed to get container status \"64c2ef9d16cd8cf15f1e6a5ef4ac7ebfe4753234b9a225c85047034b91bf4682\": rpc error: code = NotFound desc = could not find container \"64c2ef9d16cd8cf15f1e6a5ef4ac7ebfe4753234b9a225c85047034b91bf4682\": container with ID starting with 64c2ef9d16cd8cf15f1e6a5ef4ac7ebfe4753234b9a225c85047034b91bf4682 not found: ID does not exist"
Oct 30 23:02:31.613993 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:31.613965    4889 scope.go:122] "RemoveContainer" containerID="fcd12a6635b21bdca58d709888fee4cd88f8890fa114044e2004c2c4ef80c411"
Oct 30 23:02:31.614177 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:02:31.614163    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcd12a6635b21bdca58d709888fee4cd88f8890fa114044e2004c2c4ef80c411\": container with ID starting with fcd12a6635b21bdca58d709888fee4cd88f8890fa114044e2004c2c4ef80c411 not found: ID does not exist" containerID="fcd12a6635b21bdca58d709888fee4cd88f8890fa114044e2004c2c4ef80c411"
Oct 30 23:02:31.614242 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:31.614181    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcd12a6635b21bdca58d709888fee4cd88f8890fa114044e2004c2c4ef80c411"} err="failed to get container status \"fcd12a6635b21bdca58d709888fee4cd88f8890fa114044e2004c2c4ef80c411\": rpc error: code = NotFound desc = could not find container \"fcd12a6635b21bdca58d709888fee4cd88f8890fa114044e2004c2c4ef80c411\": container with ID starting with fcd12a6635b21bdca58d709888fee4cd88f8890fa114044e2004c2c4ef80c411 not found: ID does not exist"
Oct 30 23:02:31.614242 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:31.614193    4889 scope.go:122] "RemoveContainer" containerID="c31e2367c7a7cf2738859b3d7ab70c05d6dc49704f5b99bdc255d84d9efb78ca"
Oct 30 23:02:31.659622 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:31.659595    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"]
Oct 30 23:02:31.673255 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:31.673213    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-6m98h"]
Oct 30 23:02:31.704717 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:31.704688    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n"]
Oct 30 23:02:31.708304 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:31.708283    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-6b45d8f8f9-x5x2n"]
Oct 30 23:02:32.775634 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:32.775575    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d913d76-fad1-4092-bab7-e13c74cd2e0f" path="/var/lib/kubelet/pods/3d913d76-fad1-4092-bab7-e13c74cd2e0f/volumes"
Oct 30 23:02:32.776179 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:32.776163    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cfa769f-0310-4749-a929-646c9ab9bea5" path="/var/lib/kubelet/pods/7cfa769f-0310-4749-a929-646c9ab9bea5/volumes"
Oct 30 23:02:32.827013 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:32.826979    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:34.388737 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:34.388697    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:34.886675 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:34.886640    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:35.383635 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:35.383594    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:35.711052 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:35.710966    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:36.594792 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:36.594756    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs"]
Oct 30 23:02:36.616780 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:36.616747    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk"]
Oct 30 23:02:36.774491 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:36.774445    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:36.815075 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:36.815037    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52" podUID="9683180a-1f65-48bb-a62c-c78cef5799a3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:02:36.815304 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:36.815115    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt" podUID="63b2f029-b001-48d1-8c60-237096ebe16e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:02:36.950790 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:36.950711    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:37.146632 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:37.146597    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:38.106542 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:38.106509    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:38.299252 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:38.299053    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:38.659042 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:38.659002    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:38.921077 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:38.920992    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:39.966462 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:39.966440    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52"
Oct 30 23:02:39.986189 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:39.986167    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt"
Oct 30 23:02:40.066522 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.066484    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9683180a-1f65-48bb-a62c-c78cef5799a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9683180a-1f65-48bb-a62c-c78cef5799a3-plugins-config-volume\") pod \"9683180a-1f65-48bb-a62c-c78cef5799a3\" (UID: \"9683180a-1f65-48bb-a62c-c78cef5799a3\") "
Oct 30 23:02:40.067353 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.066570    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9683180a-1f65-48bb-a62c-c78cef5799a3-kube-api-access-kwhrc\" (UniqueName: \"kubernetes.io/projected/9683180a-1f65-48bb-a62c-c78cef5799a3-kube-api-access-kwhrc\") pod \"9683180a-1f65-48bb-a62c-c78cef5799a3\" (UID: \"9683180a-1f65-48bb-a62c-c78cef5799a3\") "
Oct 30 23:02:40.068103 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.068073    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9683180a-1f65-48bb-a62c-c78cef5799a3-plugins-config-volume" pod "9683180a-1f65-48bb-a62c-c78cef5799a3" (UID: "9683180a-1f65-48bb-a62c-c78cef5799a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:40.069988 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.069962    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9683180a-1f65-48bb-a62c-c78cef5799a3-kube-api-access-kwhrc" pod "9683180a-1f65-48bb-a62c-c78cef5799a3" (UID: "9683180a-1f65-48bb-a62c-c78cef5799a3"). InnerVolumeSpecName "kube-api-access-kwhrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:40.167861 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.167783    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63b2f029-b001-48d1-8c60-237096ebe16e-kube-api-access-65wt6\" (UniqueName: \"kubernetes.io/projected/63b2f029-b001-48d1-8c60-237096ebe16e-kube-api-access-65wt6\") pod \"63b2f029-b001-48d1-8c60-237096ebe16e\" (UID: \"63b2f029-b001-48d1-8c60-237096ebe16e\") "
Oct 30 23:02:40.168009 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.167947    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63b2f029-b001-48d1-8c60-237096ebe16e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b2f029-b001-48d1-8c60-237096ebe16e-plugins-config-volume\") pod \"63b2f029-b001-48d1-8c60-237096ebe16e\" (UID: \"63b2f029-b001-48d1-8c60-237096ebe16e\") "
Oct 30 23:02:40.168324 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.168294    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63b2f029-b001-48d1-8c60-237096ebe16e-plugins-config-volume" pod "63b2f029-b001-48d1-8c60-237096ebe16e" (UID: "63b2f029-b001-48d1-8c60-237096ebe16e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:40.168550 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.168533    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9683180a-1f65-48bb-a62c-c78cef5799a3-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:40.168618 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.168551    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwhrc\" (UniqueName: \"kubernetes.io/projected/9683180a-1f65-48bb-a62c-c78cef5799a3-kube-api-access-kwhrc\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:40.168618 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.168560    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b2f029-b001-48d1-8c60-237096ebe16e-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:40.170084 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.170064    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b2f029-b001-48d1-8c60-237096ebe16e-kube-api-access-65wt6" pod "63b2f029-b001-48d1-8c60-237096ebe16e" (UID: "63b2f029-b001-48d1-8c60-237096ebe16e"). InnerVolumeSpecName "kube-api-access-65wt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:40.270219 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.270173    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65wt6\" (UniqueName: \"kubernetes.io/projected/63b2f029-b001-48d1-8c60-237096ebe16e-kube-api-access-65wt6\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:40.303777 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:02:40.303739    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:02:40.426024 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.425940    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w"]
Oct 30 23:02:40.430653 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.426217    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w" podUID="457f9b6e-69a8-49d4-b65a-5afe56213f23" containerName="maas-api" containerID="cri-o://90653e59849e1d176fa765386ad14815a6eed82e88789efcc79d56129a1a3c02" gracePeriod=30
Oct 30 23:02:40.517114 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.517076    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:40.680248 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.680150    4889 generic.go:356] "Generic (PLEG): container finished" podID="63b2f029-b001-48d1-8c60-237096ebe16e" containerID="2418049e857e65cd08a191d3598901910c138abf046ca4c7101712edd89d5f03" exitCode=0
Oct 30 23:02:40.680405 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.680207    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt"
Oct 30 23:02:40.680405 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.680272    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt" event={"ID":"63b2f029-b001-48d1-8c60-237096ebe16e","Type":"ContainerDied","Data":"2418049e857e65cd08a191d3598901910c138abf046ca4c7101712edd89d5f03"}
Oct 30 23:02:40.680405 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.680330    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt" event={"ID":"63b2f029-b001-48d1-8c60-237096ebe16e","Type":"ContainerDied","Data":"0d37f7f77a96325afbdfe767e5818d3d4d7696eca9e0714e5268554a10e0859a"}
Oct 30 23:02:40.680405 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.680358    4889 scope.go:122] "RemoveContainer" containerID="2418049e857e65cd08a191d3598901910c138abf046ca4c7101712edd89d5f03"
Oct 30 23:02:40.682358 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.682332    4889 generic.go:356] "Generic (PLEG): container finished" podID="457f9b6e-69a8-49d4-b65a-5afe56213f23" containerID="90653e59849e1d176fa765386ad14815a6eed82e88789efcc79d56129a1a3c02" exitCode=0
Oct 30 23:02:40.682477 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.682410    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w" event={"ID":"457f9b6e-69a8-49d4-b65a-5afe56213f23","Type":"ContainerDied","Data":"90653e59849e1d176fa765386ad14815a6eed82e88789efcc79d56129a1a3c02"}
Oct 30 23:02:40.691926 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.691896    4889 generic.go:356] "Generic (PLEG): container finished" podID="9683180a-1f65-48bb-a62c-c78cef5799a3" containerID="3250ae08a08187eccf3a2f33c7b06aeb17c5b15ced73b0b490efe86572b354ea" exitCode=0
Oct 30 23:02:40.692024 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.691983    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52" event={"ID":"9683180a-1f65-48bb-a62c-c78cef5799a3","Type":"ContainerDied","Data":"3250ae08a08187eccf3a2f33c7b06aeb17c5b15ced73b0b490efe86572b354ea"}
Oct 30 23:02:40.692074 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.692033    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52" event={"ID":"9683180a-1f65-48bb-a62c-c78cef5799a3","Type":"ContainerDied","Data":"e03366a02d5b1fd31a33ab6b69505e4d9daec3bda4811c91cc3d4edffe17bfcd"}
Oct 30 23:02:40.692149 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.692135    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52"
Oct 30 23:02:40.693291 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.693273    4889 scope.go:122] "RemoveContainer" containerID="2418049e857e65cd08a191d3598901910c138abf046ca4c7101712edd89d5f03"
Oct 30 23:02:40.693620 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:02:40.693581    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2418049e857e65cd08a191d3598901910c138abf046ca4c7101712edd89d5f03\": container with ID starting with 2418049e857e65cd08a191d3598901910c138abf046ca4c7101712edd89d5f03 not found: ID does not exist" containerID="2418049e857e65cd08a191d3598901910c138abf046ca4c7101712edd89d5f03"
Oct 30 23:02:40.693747 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.693630    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2418049e857e65cd08a191d3598901910c138abf046ca4c7101712edd89d5f03"} err="failed to get container status \"2418049e857e65cd08a191d3598901910c138abf046ca4c7101712edd89d5f03\": rpc error: code = NotFound desc = could not find container \"2418049e857e65cd08a191d3598901910c138abf046ca4c7101712edd89d5f03\": container with ID starting with 2418049e857e65cd08a191d3598901910c138abf046ca4c7101712edd89d5f03 not found: ID does not exist"
Oct 30 23:02:40.693747 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.693653    4889 scope.go:122] "RemoveContainer" containerID="3250ae08a08187eccf3a2f33c7b06aeb17c5b15ced73b0b490efe86572b354ea"
Oct 30 23:02:40.709086 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.709068    4889 scope.go:122] "RemoveContainer" containerID="3250ae08a08187eccf3a2f33c7b06aeb17c5b15ced73b0b490efe86572b354ea"
Oct 30 23:02:40.709490 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:02:40.709461    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3250ae08a08187eccf3a2f33c7b06aeb17c5b15ced73b0b490efe86572b354ea\": container with ID starting with 3250ae08a08187eccf3a2f33c7b06aeb17c5b15ced73b0b490efe86572b354ea not found: ID does not exist" containerID="3250ae08a08187eccf3a2f33c7b06aeb17c5b15ced73b0b490efe86572b354ea"
Oct 30 23:02:40.709595 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.709503    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3250ae08a08187eccf3a2f33c7b06aeb17c5b15ced73b0b490efe86572b354ea"} err="failed to get container status \"3250ae08a08187eccf3a2f33c7b06aeb17c5b15ced73b0b490efe86572b354ea\": rpc error: code = NotFound desc = could not find container \"3250ae08a08187eccf3a2f33c7b06aeb17c5b15ced73b0b490efe86572b354ea\": container with ID starting with 3250ae08a08187eccf3a2f33c7b06aeb17c5b15ced73b0b490efe86572b354ea not found: ID does not exist"
Oct 30 23:02:40.730147 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.730124    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:40.744164 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.744141    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w"
Oct 30 23:02:40.755300 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.755280    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt"]
Oct 30 23:02:40.758583 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.758565    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0ff5b0-6f4dc7d5f-pmqnt"]
Oct 30 23:02:40.775496 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.775473    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63b2f029-b001-48d1-8c60-237096ebe16e" path="/var/lib/kubelet/pods/63b2f029-b001-48d1-8c60-237096ebe16e/volumes"
Oct 30 23:02:40.837903 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.837872    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52"]
Oct 30 23:02:40.841343 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.841315    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-0ff5b0-7c4b7489d7-xlk52"]
Oct 30 23:02:40.877514 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.877478    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/457f9b6e-69a8-49d4-b65a-5afe56213f23-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/457f9b6e-69a8-49d4-b65a-5afe56213f23-maas-api-tls\") pod \"457f9b6e-69a8-49d4-b65a-5afe56213f23\" (UID: \"457f9b6e-69a8-49d4-b65a-5afe56213f23\") "
Oct 30 23:02:40.877707 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.877585    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/457f9b6e-69a8-49d4-b65a-5afe56213f23-kube-api-access-xdtr5\" (UniqueName: \"kubernetes.io/projected/457f9b6e-69a8-49d4-b65a-5afe56213f23-kube-api-access-xdtr5\") pod \"457f9b6e-69a8-49d4-b65a-5afe56213f23\" (UID: \"457f9b6e-69a8-49d4-b65a-5afe56213f23\") "
Oct 30 23:02:40.877775 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.877750    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/457f9b6e-69a8-49d4-b65a-5afe56213f23-metrics-certs\" (UniqueName: \"kubernetes.io/secret/457f9b6e-69a8-49d4-b65a-5afe56213f23-metrics-certs\") pod \"457f9b6e-69a8-49d4-b65a-5afe56213f23\" (UID: \"457f9b6e-69a8-49d4-b65a-5afe56213f23\") "
Oct 30 23:02:40.877845 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.877829    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/457f9b6e-69a8-49d4-b65a-5afe56213f23-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/457f9b6e-69a8-49d4-b65a-5afe56213f23-openshift-service-ca\") pod \"457f9b6e-69a8-49d4-b65a-5afe56213f23\" (UID: \"457f9b6e-69a8-49d4-b65a-5afe56213f23\") "
Oct 30 23:02:40.878252 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.878202    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/457f9b6e-69a8-49d4-b65a-5afe56213f23-openshift-service-ca" pod "457f9b6e-69a8-49d4-b65a-5afe56213f23" (UID: "457f9b6e-69a8-49d4-b65a-5afe56213f23"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:40.879090 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.879065    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/457f9b6e-69a8-49d4-b65a-5afe56213f23-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:40.880285 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.880241    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457f9b6e-69a8-49d4-b65a-5afe56213f23-kube-api-access-xdtr5" pod "457f9b6e-69a8-49d4-b65a-5afe56213f23" (UID: "457f9b6e-69a8-49d4-b65a-5afe56213f23"). InnerVolumeSpecName "kube-api-access-xdtr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:40.880432 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.880415    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/457f9b6e-69a8-49d4-b65a-5afe56213f23-metrics-certs" pod "457f9b6e-69a8-49d4-b65a-5afe56213f23" (UID: "457f9b6e-69a8-49d4-b65a-5afe56213f23"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:02:40.880492 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.880455    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/457f9b6e-69a8-49d4-b65a-5afe56213f23-maas-api-tls" pod "457f9b6e-69a8-49d4-b65a-5afe56213f23" (UID: "457f9b6e-69a8-49d4-b65a-5afe56213f23"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:02:40.893384 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:40.893350    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:41.027244 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:41.027091    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/457f9b6e-69a8-49d4-b65a-5afe56213f23-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:41.027244 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:41.027125    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdtr5\" (UniqueName: \"kubernetes.io/projected/457f9b6e-69a8-49d4-b65a-5afe56213f23-kube-api-access-xdtr5\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:41.027244 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:41.027139    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/457f9b6e-69a8-49d4-b65a-5afe56213f23-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:41.218882 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:41.218850    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:41.385484 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:41.385385    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:41.596033 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:41.595997    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs" podUID="94b31d20-a647-435f-8cdb-9d884c864207" containerName="payload-processing" containerID="cri-o://5f48b4eaa5b692860fc67f9265ef6cad9934cac8c3ad78583111a9c62cb38a51" gracePeriod=25
Oct 30 23:02:41.617354 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:41.617320    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk" podUID="1511ead8-a673-47c2-9c21-9078d9e051ed" containerName="payload-pre-processing" containerID="cri-o://79e7d3f65230f8769b2887a5af8e8c347bf1ebbb5b73f0b1139ac30f24ed4f9c" gracePeriod=25
Oct 30 23:02:41.703215 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:41.703128    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w" event={"ID":"457f9b6e-69a8-49d4-b65a-5afe56213f23","Type":"ContainerDied","Data":"462963f02d408952bfc782762c7226d2f35a2ee926424fea0734cd29f1862802"}
Oct 30 23:02:41.703215 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:41.703148    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w"
Oct 30 23:02:41.703215 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:41.703195    4889 scope.go:122] "RemoveContainer" containerID="90653e59849e1d176fa765386ad14815a6eed82e88789efcc79d56129a1a3c02"
Oct 30 23:02:41.759405 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:41.759377    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w"]
Oct 30 23:02:41.773075 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:41.773052    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88979-dddf57697-fjj4w"]
Oct 30 23:02:41.783833 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:41.783804    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:42.326147 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:42.326107    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:42.498590 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:42.498556    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:42.618673 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:42.602470    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"]
Oct 30 23:02:42.618673 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:42.602760    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d" podUID="fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4" containerName="istio-proxy" containerID="cri-o://69e6b71e4c80c0aa052106f1e0649b85eee418eefe625fcfc8d37c1716a080f7" gracePeriod=30
Oct 30 23:02:42.683503 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:42.683464    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:42.800942 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:42.800885    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="457f9b6e-69a8-49d4-b65a-5afe56213f23" path="/var/lib/kubelet/pods/457f9b6e-69a8-49d4-b65a-5afe56213f23/volumes"
Oct 30 23:02:42.801708 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:42.801680    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9683180a-1f65-48bb-a62c-c78cef5799a3" path="/var/lib/kubelet/pods/9683180a-1f65-48bb-a62c-c78cef5799a3/volumes"
Oct 30 23:02:43.150754 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:43.150708    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:45.703897 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:45.703862    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:46.198369 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:46.198334    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs" podUID="94b31d20-a647-435f-8cdb-9d884c864207" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:02:46.198563 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:46.198421    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk" podUID="1511ead8-a673-47c2-9c21-9078d9e051ed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:02:47.533448 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:47.533354    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:47.849260 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:47.849134    4889 generic.go:356] "Generic (PLEG): container finished" podID="fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4" containerID="69e6b71e4c80c0aa052106f1e0649b85eee418eefe625fcfc8d37c1716a080f7" exitCode=0
Oct 30 23:02:47.849260 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:47.849217    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d" event={"ID":"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4","Type":"ContainerDied","Data":"69e6b71e4c80c0aa052106f1e0649b85eee418eefe625fcfc8d37c1716a080f7"}
Oct 30 23:02:47.884275 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:47.884242    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 23:02:47.940720 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:47.940683    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-token\" (UniqueName: \"kubernetes.io/projected/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-token\") pod \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") "
Oct 30 23:02:47.941034 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:47.941008    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istiod-ca-cert\") pod \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") "
Oct 30 23:02:47.941272 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:47.941253    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-envoy\") pod \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") "
Oct 30 23:02:47.941418 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:47.941393    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-credential-socket\") pod \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") "
Oct 30 23:02:47.941595 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:47.941449    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-data\") pod \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") "
Oct 30 23:02:47.941687 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:47.941591    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-podinfo\") pod \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") "
Oct 30 23:02:47.941687 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:47.941652    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-workload-socket\") pod \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") "
Oct 30 23:02:47.941852 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:47.941750    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-workload-certs\") pod \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") "
Oct 30 23:02:47.941941 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:47.941857    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-kube-api-access-pwb4z\" (UniqueName: \"kubernetes.io/projected/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-kube-api-access-pwb4z\") pod \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\" (UID: \"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4\") "
Oct 30 23:02:47.941941 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:47.941895    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istiod-ca-cert" pod "fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4" (UID: "fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:47.942482 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:47.942461    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:47.943507 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:47.943440    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-data" pod "fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4" (UID: "fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:47.943507 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:47.943481    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-workload-socket" pod "fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4" (UID: "fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:47.943722 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:47.943552    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-credential-socket" pod "fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4" (UID: "fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:47.947881 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:47.944909    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-token" pod "fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4" (UID: "fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:47.947881 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:47.946384    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-envoy" pod "fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4" (UID: "fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:47.947881 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:47.946451    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-podinfo" pod "fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4" (UID: "fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:02:47.948643 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:47.948384    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-kube-api-access-pwb4z" pod "fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4" (UID: "fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4"). InnerVolumeSpecName "kube-api-access-pwb4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:47.948643 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:47.948479    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-workload-certs" pod "fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4" (UID: "fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:48.044358 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:48.044326    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:48.044358 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:48.044352    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:48.044358 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:48.044361    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:48.044593 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:48.044370    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:48.044593 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:48.044380    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:48.044593 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:48.044390    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:48.044593 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:48.044398    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:48.044593 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:48.044409    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwb4z\" (UniqueName: \"kubernetes.io/projected/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4-kube-api-access-pwb4z\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:48.270196 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:48.270158    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:48.462005 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:48.461960    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:48.516002 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:48.515961    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"]
Oct 30 23:02:48.516521 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:48.516489    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g" podUID="d2279298-a1a4-469d-a188-f41478b6a598" containerName="istio-proxy" containerID="cri-o://2dd5d7e1811bbf9f5aeff1b49ba3e29f8dd3c016922fa6acf6b1f86fe5cc68af" gracePeriod=30
Oct 30 23:02:48.659898 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:48.659801    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:48.801069 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:48.801038    4889 patch_prober.go:29] interesting pod/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:02:48.801264 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:48.801107    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g" podUID="d2279298-a1a4-469d-a188-f41478b6a598" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:02:48.857662 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:48.857635    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"
Oct 30 23:02:48.857662 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:48.857653    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d" event={"ID":"fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4","Type":"ContainerDied","Data":"76d624fb03dd61f8eda18203d86b3b7b822d18a9055e933398799e4ef3149398"}
Oct 30 23:02:48.857892 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:48.857713    4889 scope.go:122] "RemoveContainer" containerID="69e6b71e4c80c0aa052106f1e0649b85eee418eefe625fcfc8d37c1716a080f7"
Oct 30 23:02:48.915142 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:48.915072    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"]
Oct 30 23:02:48.927612 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:48.927586    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-9ps8d"]
Oct 30 23:02:49.528240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:49.528183    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:50.776542 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:50.776508    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4" path="/var/lib/kubelet/pods/fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4/volumes"
Oct 30 23:02:51.596453 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:51.596415    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:51.848750 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:51.848663    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:53.620055 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:53.620019    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm"]
Oct 30 23:02:53.655851 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:53.655819    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995"]
Oct 30 23:02:53.798532 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:53.798442    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:53.847865 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:53.847831    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 23:02:53.899462 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:53.899426    4889 generic.go:356] "Generic (PLEG): container finished" podID="d2279298-a1a4-469d-a188-f41478b6a598" containerID="2dd5d7e1811bbf9f5aeff1b49ba3e29f8dd3c016922fa6acf6b1f86fe5cc68af" exitCode=0
Oct 30 23:02:53.899643 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:53.899481    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"
Oct 30 23:02:53.899643 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:53.899479    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g" event={"ID":"d2279298-a1a4-469d-a188-f41478b6a598","Type":"ContainerDied","Data":"2dd5d7e1811bbf9f5aeff1b49ba3e29f8dd3c016922fa6acf6b1f86fe5cc68af"}
Oct 30 23:02:53.899643 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:53.899603    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g" event={"ID":"d2279298-a1a4-469d-a188-f41478b6a598","Type":"ContainerDied","Data":"8f2c8aebc65063ce1fb6a2939518279beb518f54cd6568f9d8f48ced225288d7"}
Oct 30 23:02:53.899643 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:53.899626    4889 scope.go:122] "RemoveContainer" containerID="2dd5d7e1811bbf9f5aeff1b49ba3e29f8dd3c016922fa6acf6b1f86fe5cc68af"
Oct 30 23:02:53.912688 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:53.912660    4889 scope.go:122] "RemoveContainer" containerID="9289fcccac2f5ebe83a35605eccd354110ae8e9ebd91bab76186f3593754995e"
Oct 30 23:02:53.925943 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:53.925919    4889 scope.go:122] "RemoveContainer" containerID="2dd5d7e1811bbf9f5aeff1b49ba3e29f8dd3c016922fa6acf6b1f86fe5cc68af"
Oct 30 23:02:53.926377 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:02:53.926347    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd5d7e1811bbf9f5aeff1b49ba3e29f8dd3c016922fa6acf6b1f86fe5cc68af\": container with ID starting with 2dd5d7e1811bbf9f5aeff1b49ba3e29f8dd3c016922fa6acf6b1f86fe5cc68af not found: ID does not exist" containerID="2dd5d7e1811bbf9f5aeff1b49ba3e29f8dd3c016922fa6acf6b1f86fe5cc68af"
Oct 30 23:02:53.926465 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:53.926388    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd5d7e1811bbf9f5aeff1b49ba3e29f8dd3c016922fa6acf6b1f86fe5cc68af"} err="failed to get container status \"2dd5d7e1811bbf9f5aeff1b49ba3e29f8dd3c016922fa6acf6b1f86fe5cc68af\": rpc error: code = NotFound desc = could not find container \"2dd5d7e1811bbf9f5aeff1b49ba3e29f8dd3c016922fa6acf6b1f86fe5cc68af\": container with ID starting with 2dd5d7e1811bbf9f5aeff1b49ba3e29f8dd3c016922fa6acf6b1f86fe5cc68af not found: ID does not exist"
Oct 30 23:02:53.926465 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:53.926414    4889 scope.go:122] "RemoveContainer" containerID="9289fcccac2f5ebe83a35605eccd354110ae8e9ebd91bab76186f3593754995e"
Oct 30 23:02:53.926786 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:02:53.926760    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9289fcccac2f5ebe83a35605eccd354110ae8e9ebd91bab76186f3593754995e\": container with ID starting with 9289fcccac2f5ebe83a35605eccd354110ae8e9ebd91bab76186f3593754995e not found: ID does not exist" containerID="9289fcccac2f5ebe83a35605eccd354110ae8e9ebd91bab76186f3593754995e"
Oct 30 23:02:53.926904 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:53.926798    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9289fcccac2f5ebe83a35605eccd354110ae8e9ebd91bab76186f3593754995e"} err="failed to get container status \"9289fcccac2f5ebe83a35605eccd354110ae8e9ebd91bab76186f3593754995e\": rpc error: code = NotFound desc = could not find container \"9289fcccac2f5ebe83a35605eccd354110ae8e9ebd91bab76186f3593754995e\": container with ID starting with 9289fcccac2f5ebe83a35605eccd354110ae8e9ebd91bab76186f3593754995e not found: ID does not exist"
Oct 30 23:02:54.023935 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.023895    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:54.036022 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.035991    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2279298-a1a4-469d-a188-f41478b6a598-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2279298-a1a4-469d-a188-f41478b6a598-istiod-ca-cert\") pod \"d2279298-a1a4-469d-a188-f41478b6a598\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") "
Oct 30 23:02:54.036022 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.036025    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2279298-a1a4-469d-a188-f41478b6a598-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2279298-a1a4-469d-a188-f41478b6a598-istio-podinfo\") pod \"d2279298-a1a4-469d-a188-f41478b6a598\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") "
Oct 30 23:02:54.036200 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.036048    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2279298-a1a4-469d-a188-f41478b6a598-istio-token\" (UniqueName: \"kubernetes.io/projected/d2279298-a1a4-469d-a188-f41478b6a598-istio-token\") pod \"d2279298-a1a4-469d-a188-f41478b6a598\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") "
Oct 30 23:02:54.036200 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.036144    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-istio-data\") pod \"d2279298-a1a4-469d-a188-f41478b6a598\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") "
Oct 30 23:02:54.036360 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.036275    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-istio-envoy\") pod \"d2279298-a1a4-469d-a188-f41478b6a598\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") "
Oct 30 23:02:54.036360 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.036333    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-workload-certs\") pod \"d2279298-a1a4-469d-a188-f41478b6a598\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") "
Oct 30 23:02:54.036477 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.036457    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-credential-socket\") pod \"d2279298-a1a4-469d-a188-f41478b6a598\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") "
Oct 30 23:02:54.036534 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.036482    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2279298-a1a4-469d-a188-f41478b6a598-istiod-ca-cert" pod "d2279298-a1a4-469d-a188-f41478b6a598" (UID: "d2279298-a1a4-469d-a188-f41478b6a598"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:54.036534 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.036506    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2279298-a1a4-469d-a188-f41478b6a598-kube-api-access-vns6f\" (UniqueName: \"kubernetes.io/projected/d2279298-a1a4-469d-a188-f41478b6a598-kube-api-access-vns6f\") pod \"d2279298-a1a4-469d-a188-f41478b6a598\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") "
Oct 30 23:02:54.036658 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.036594    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-workload-socket\") pod \"d2279298-a1a4-469d-a188-f41478b6a598\" (UID: \"d2279298-a1a4-469d-a188-f41478b6a598\") "
Oct 30 23:02:54.036833 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.036588    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-workload-certs" pod "d2279298-a1a4-469d-a188-f41478b6a598" (UID: "d2279298-a1a4-469d-a188-f41478b6a598"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:54.036833 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.036693    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-credential-socket" pod "d2279298-a1a4-469d-a188-f41478b6a598" (UID: "d2279298-a1a4-469d-a188-f41478b6a598"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:54.037029 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.036770    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-istio-data" pod "d2279298-a1a4-469d-a188-f41478b6a598" (UID: "d2279298-a1a4-469d-a188-f41478b6a598"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:54.037029 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.036780    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-workload-socket" pod "d2279298-a1a4-469d-a188-f41478b6a598" (UID: "d2279298-a1a4-469d-a188-f41478b6a598"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:54.037207 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.037188    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:54.037314 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.037212    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:54.037314 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.037243    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:54.037314 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.037256    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:54.037314 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.037270    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2279298-a1a4-469d-a188-f41478b6a598-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:54.038647 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.038623    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2279298-a1a4-469d-a188-f41478b6a598-istio-podinfo" pod "d2279298-a1a4-469d-a188-f41478b6a598" (UID: "d2279298-a1a4-469d-a188-f41478b6a598"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:02:54.038749 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.038662    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2279298-a1a4-469d-a188-f41478b6a598-istio-token" pod "d2279298-a1a4-469d-a188-f41478b6a598" (UID: "d2279298-a1a4-469d-a188-f41478b6a598"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:54.039178 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.039155    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-istio-envoy" pod "d2279298-a1a4-469d-a188-f41478b6a598" (UID: "d2279298-a1a4-469d-a188-f41478b6a598"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:54.039292 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.039256    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2279298-a1a4-469d-a188-f41478b6a598-kube-api-access-vns6f" pod "d2279298-a1a4-469d-a188-f41478b6a598" (UID: "d2279298-a1a4-469d-a188-f41478b6a598"). InnerVolumeSpecName "kube-api-access-vns6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:54.138190 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.138161    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vns6f\" (UniqueName: \"kubernetes.io/projected/d2279298-a1a4-469d-a188-f41478b6a598-kube-api-access-vns6f\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:54.138190 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.138190    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2279298-a1a4-469d-a188-f41478b6a598-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:54.138363 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.138199    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2279298-a1a4-469d-a188-f41478b6a598-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:54.138363 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.138208    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2279298-a1a4-469d-a188-f41478b6a598-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:02:54.294061 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.294019    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"]
Oct 30 23:02:54.304803 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.304718    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f88979-openshift-default-674df6d98f-tr49g"]
Oct 30 23:02:54.776672 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:54.776643    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2279298-a1a4-469d-a188-f41478b6a598" path="/var/lib/kubelet/pods/d2279298-a1a4-469d-a188-f41478b6a598/volumes"
Oct 30 23:02:55.794953 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:55.794914    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:56.198023 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:56.197939    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs" podUID="94b31d20-a647-435f-8cdb-9d884c864207" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:02:56.198180 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:56.197948    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk" podUID="1511ead8-a673-47c2-9c21-9078d9e051ed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:02:57.536401 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:57.536360    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:58.621215 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:58.621177    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm" podUID="2ace3d4c-11b6-41ca-a276-37746b3791fa" containerName="payload-processing" containerID="cri-o://e42435c4b1d5891e13096494a67eb64caeb516f387614131dfd299be6d59616d" gracePeriod=25
Oct 30 23:02:58.657095 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:58.657048    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995" podUID="e0a01454-5acd-417b-9e3c-62c826ef6558" containerName="payload-pre-processing" containerID="cri-o://fcb7050194b24fe2cfd5e1eda326c9a02a60a76014c202a89a1d592c48fcd3dd" gracePeriod=25
Oct 30 23:02:59.325145 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:02:59.325104    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:01.248240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:01.248180    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:01.923626 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:01.923586    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs"
Oct 30 23:03:01.963164 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:01.963127    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk"
Oct 30 23:03:01.963365 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:01.963273    4889 generic.go:356] "Generic (PLEG): container finished" podID="94b31d20-a647-435f-8cdb-9d884c864207" containerID="5f48b4eaa5b692860fc67f9265ef6cad9934cac8c3ad78583111a9c62cb38a51" exitCode=0
Oct 30 23:03:01.963365 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:01.963357    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs"
Oct 30 23:03:01.963510 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:01.963378    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs" event={"ID":"94b31d20-a647-435f-8cdb-9d884c864207","Type":"ContainerDied","Data":"5f48b4eaa5b692860fc67f9265ef6cad9934cac8c3ad78583111a9c62cb38a51"}
Oct 30 23:03:01.963510 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:01.963476    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs" event={"ID":"94b31d20-a647-435f-8cdb-9d884c864207","Type":"ContainerDied","Data":"ca05f80e583050c5fbb62daf6a61f14178a046fcd3a790b60792640e4cebcba1"}
Oct 30 23:03:01.963628 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:01.963510    4889 scope.go:122] "RemoveContainer" containerID="5f48b4eaa5b692860fc67f9265ef6cad9934cac8c3ad78583111a9c62cb38a51"
Oct 30 23:03:01.964778 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:01.964756    4889 generic.go:356] "Generic (PLEG): container finished" podID="1511ead8-a673-47c2-9c21-9078d9e051ed" containerID="79e7d3f65230f8769b2887a5af8e8c347bf1ebbb5b73f0b1139ac30f24ed4f9c" exitCode=0
Oct 30 23:03:01.964916 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:01.964823    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk" event={"ID":"1511ead8-a673-47c2-9c21-9078d9e051ed","Type":"ContainerDied","Data":"79e7d3f65230f8769b2887a5af8e8c347bf1ebbb5b73f0b1139ac30f24ed4f9c"}
Oct 30 23:03:01.964916 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:01.964856    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk" event={"ID":"1511ead8-a673-47c2-9c21-9078d9e051ed","Type":"ContainerDied","Data":"20bd03ff9056dec08b225b7f4df97b0d8c5eddd1c812156231303ce00c17a0a9"}
Oct 30 23:03:01.964916 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:01.964855    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk"
Oct 30 23:03:01.970009 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:01.969433    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94b31d20-a647-435f-8cdb-9d884c864207-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94b31d20-a647-435f-8cdb-9d884c864207-plugins-config-volume\") pod \"94b31d20-a647-435f-8cdb-9d884c864207\" (UID: \"94b31d20-a647-435f-8cdb-9d884c864207\") "
Oct 30 23:03:01.970009 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:01.969567    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94b31d20-a647-435f-8cdb-9d884c864207-kube-api-access-dfgdp\" (UniqueName: \"kubernetes.io/projected/94b31d20-a647-435f-8cdb-9d884c864207-kube-api-access-dfgdp\") pod \"94b31d20-a647-435f-8cdb-9d884c864207\" (UID: \"94b31d20-a647-435f-8cdb-9d884c864207\") "
Oct 30 23:03:01.972750 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:01.972727    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b31d20-a647-435f-8cdb-9d884c864207-plugins-config-volume" pod "94b31d20-a647-435f-8cdb-9d884c864207" (UID: "94b31d20-a647-435f-8cdb-9d884c864207"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:01.977777 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:01.977752    4889 scope.go:122] "RemoveContainer" containerID="5f48b4eaa5b692860fc67f9265ef6cad9934cac8c3ad78583111a9c62cb38a51"
Oct 30 23:03:01.978149 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:03:01.978115    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f48b4eaa5b692860fc67f9265ef6cad9934cac8c3ad78583111a9c62cb38a51\": container with ID starting with 5f48b4eaa5b692860fc67f9265ef6cad9934cac8c3ad78583111a9c62cb38a51 not found: ID does not exist" containerID="5f48b4eaa5b692860fc67f9265ef6cad9934cac8c3ad78583111a9c62cb38a51"
Oct 30 23:03:01.978289 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:01.978159    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f48b4eaa5b692860fc67f9265ef6cad9934cac8c3ad78583111a9c62cb38a51"} err="failed to get container status \"5f48b4eaa5b692860fc67f9265ef6cad9934cac8c3ad78583111a9c62cb38a51\": rpc error: code = NotFound desc = could not find container \"5f48b4eaa5b692860fc67f9265ef6cad9934cac8c3ad78583111a9c62cb38a51\": container with ID starting with 5f48b4eaa5b692860fc67f9265ef6cad9934cac8c3ad78583111a9c62cb38a51 not found: ID does not exist"
Oct 30 23:03:01.978289 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:01.978177    4889 scope.go:122] "RemoveContainer" containerID="79e7d3f65230f8769b2887a5af8e8c347bf1ebbb5b73f0b1139ac30f24ed4f9c"
Oct 30 23:03:01.980663 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:01.980641    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b31d20-a647-435f-8cdb-9d884c864207-kube-api-access-dfgdp" pod "94b31d20-a647-435f-8cdb-9d884c864207" (UID: "94b31d20-a647-435f-8cdb-9d884c864207"). InnerVolumeSpecName "kube-api-access-dfgdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:01.992693 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:01.992596    4889 scope.go:122] "RemoveContainer" containerID="79e7d3f65230f8769b2887a5af8e8c347bf1ebbb5b73f0b1139ac30f24ed4f9c"
Oct 30 23:03:01.993098 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:03:01.993072    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e7d3f65230f8769b2887a5af8e8c347bf1ebbb5b73f0b1139ac30f24ed4f9c\": container with ID starting with 79e7d3f65230f8769b2887a5af8e8c347bf1ebbb5b73f0b1139ac30f24ed4f9c not found: ID does not exist" containerID="79e7d3f65230f8769b2887a5af8e8c347bf1ebbb5b73f0b1139ac30f24ed4f9c"
Oct 30 23:03:01.993183 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:01.993111    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e7d3f65230f8769b2887a5af8e8c347bf1ebbb5b73f0b1139ac30f24ed4f9c"} err="failed to get container status \"79e7d3f65230f8769b2887a5af8e8c347bf1ebbb5b73f0b1139ac30f24ed4f9c\": rpc error: code = NotFound desc = could not find container \"79e7d3f65230f8769b2887a5af8e8c347bf1ebbb5b73f0b1139ac30f24ed4f9c\": container with ID starting with 79e7d3f65230f8769b2887a5af8e8c347bf1ebbb5b73f0b1139ac30f24ed4f9c not found: ID does not exist"
Oct 30 23:03:02.071170 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.071134    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1511ead8-a673-47c2-9c21-9078d9e051ed-kube-api-access-nhs7h\" (UniqueName: \"kubernetes.io/projected/1511ead8-a673-47c2-9c21-9078d9e051ed-kube-api-access-nhs7h\") pod \"1511ead8-a673-47c2-9c21-9078d9e051ed\" (UID: \"1511ead8-a673-47c2-9c21-9078d9e051ed\") "
Oct 30 23:03:02.071560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.071539    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1511ead8-a673-47c2-9c21-9078d9e051ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1511ead8-a673-47c2-9c21-9078d9e051ed-plugins-config-volume\") pod \"1511ead8-a673-47c2-9c21-9078d9e051ed\" (UID: \"1511ead8-a673-47c2-9c21-9078d9e051ed\") "
Oct 30 23:03:02.071999 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.071973    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1511ead8-a673-47c2-9c21-9078d9e051ed-plugins-config-volume" pod "1511ead8-a673-47c2-9c21-9078d9e051ed" (UID: "1511ead8-a673-47c2-9c21-9078d9e051ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:02.072090 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.071977    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94b31d20-a647-435f-8cdb-9d884c864207-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:02.072090 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.072037    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfgdp\" (UniqueName: \"kubernetes.io/projected/94b31d20-a647-435f-8cdb-9d884c864207-kube-api-access-dfgdp\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:02.074729 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.074699    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1511ead8-a673-47c2-9c21-9078d9e051ed-kube-api-access-nhs7h" pod "1511ead8-a673-47c2-9c21-9078d9e051ed" (UID: "1511ead8-a673-47c2-9c21-9078d9e051ed"). InnerVolumeSpecName "kube-api-access-nhs7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:02.173624 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.173547    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1511ead8-a673-47c2-9c21-9078d9e051ed-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:02.173624 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.173573    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhs7h\" (UniqueName: \"kubernetes.io/projected/1511ead8-a673-47c2-9c21-9078d9e051ed-kube-api-access-nhs7h\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:02.322247 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.319818    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs"]
Oct 30 23:03:02.331416 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.331376    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-d9f1c2-6475ccbbb9-x7xgs"]
Oct 30 23:03:02.374369 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.374336    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk"]
Oct 30 23:03:02.379529 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.379500    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-d9f1c2-5bf66857f-zbxhk"]
Oct 30 23:03:02.435689 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.435609    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:02.485187 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.485152    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf"]
Oct 30 23:03:02.485405 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.485382    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf" podUID="c27dca2f-2be7-4a83-bdc0-9df7082f0027" containerName="payload-processing" containerID="cri-o://1dcc57b08841bea7db75a2463a5430763d2a3418aca74e6737ba21eaed02d97d" gracePeriod=30
Oct 30 23:03:02.620747 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.620684    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:02.776085 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.776051    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1511ead8-a673-47c2-9c21-9078d9e051ed" path="/var/lib/kubelet/pods/1511ead8-a673-47c2-9c21-9078d9e051ed/volumes"
Oct 30 23:03:02.776533 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.776517    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94b31d20-a647-435f-8cdb-9d884c864207" path="/var/lib/kubelet/pods/94b31d20-a647-435f-8cdb-9d884c864207/volumes"
Oct 30 23:03:02.807363 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.802996    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf"
Oct 30 23:03:02.825616 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.824816    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:02.893514 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.888342    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c27dca2f-2be7-4a83-bdc0-9df7082f0027-kube-api-access-pjhsl\" (UniqueName: \"kubernetes.io/projected/c27dca2f-2be7-4a83-bdc0-9df7082f0027-kube-api-access-pjhsl\") pod \"c27dca2f-2be7-4a83-bdc0-9df7082f0027\" (UID: \"c27dca2f-2be7-4a83-bdc0-9df7082f0027\") "
Oct 30 23:03:02.893514 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.888399    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c27dca2f-2be7-4a83-bdc0-9df7082f0027-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c27dca2f-2be7-4a83-bdc0-9df7082f0027-plugins-config-volume\") pod \"c27dca2f-2be7-4a83-bdc0-9df7082f0027\" (UID: \"c27dca2f-2be7-4a83-bdc0-9df7082f0027\") "
Oct 30 23:03:02.893514 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.890759    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c27dca2f-2be7-4a83-bdc0-9df7082f0027-plugins-config-volume" pod "c27dca2f-2be7-4a83-bdc0-9df7082f0027" (UID: "c27dca2f-2be7-4a83-bdc0-9df7082f0027"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:02.901516 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.901480    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27dca2f-2be7-4a83-bdc0-9df7082f0027-kube-api-access-pjhsl" pod "c27dca2f-2be7-4a83-bdc0-9df7082f0027" (UID: "c27dca2f-2be7-4a83-bdc0-9df7082f0027"). InnerVolumeSpecName "kube-api-access-pjhsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:02.974761 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.974726    4889 generic.go:356] "Generic (PLEG): container finished" podID="c27dca2f-2be7-4a83-bdc0-9df7082f0027" containerID="1dcc57b08841bea7db75a2463a5430763d2a3418aca74e6737ba21eaed02d97d" exitCode=0
Oct 30 23:03:02.974960 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.974778    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf"
Oct 30 23:03:02.974960 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.974811    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf" event={"ID":"c27dca2f-2be7-4a83-bdc0-9df7082f0027","Type":"ContainerDied","Data":"1dcc57b08841bea7db75a2463a5430763d2a3418aca74e6737ba21eaed02d97d"}
Oct 30 23:03:02.974960 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.974874    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf" event={"ID":"c27dca2f-2be7-4a83-bdc0-9df7082f0027","Type":"ContainerDied","Data":"90b0ab90d45cecfa015dda53a898b4bc7d408eae3301b392350b5b734873c405"}
Oct 30 23:03:02.974960 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.974904    4889 scope.go:122] "RemoveContainer" containerID="1dcc57b08841bea7db75a2463a5430763d2a3418aca74e6737ba21eaed02d97d"
Oct 30 23:03:02.986866 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.986845    4889 scope.go:122] "RemoveContainer" containerID="1dcc57b08841bea7db75a2463a5430763d2a3418aca74e6737ba21eaed02d97d"
Oct 30 23:03:02.987200 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:03:02.987180    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dcc57b08841bea7db75a2463a5430763d2a3418aca74e6737ba21eaed02d97d\": container with ID starting with 1dcc57b08841bea7db75a2463a5430763d2a3418aca74e6737ba21eaed02d97d not found: ID does not exist" containerID="1dcc57b08841bea7db75a2463a5430763d2a3418aca74e6737ba21eaed02d97d"
Oct 30 23:03:02.987344 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.987208    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dcc57b08841bea7db75a2463a5430763d2a3418aca74e6737ba21eaed02d97d"} err="failed to get container status \"1dcc57b08841bea7db75a2463a5430763d2a3418aca74e6737ba21eaed02d97d\": rpc error: code = NotFound desc = could not find container \"1dcc57b08841bea7db75a2463a5430763d2a3418aca74e6737ba21eaed02d97d\": container with ID starting with 1dcc57b08841bea7db75a2463a5430763d2a3418aca74e6737ba21eaed02d97d not found: ID does not exist"
Oct 30 23:03:02.990184 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.990157    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjhsl\" (UniqueName: \"kubernetes.io/projected/c27dca2f-2be7-4a83-bdc0-9df7082f0027-kube-api-access-pjhsl\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:02.990344 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:02.990188    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c27dca2f-2be7-4a83-bdc0-9df7082f0027-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:03.035163 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:03.035077    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf"]
Oct 30 23:03:03.038685 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:03.038663    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cdfa86-79959f8c88-hcpjf"]
Oct 30 23:03:03.053590 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:03.053558    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:03.222273 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:03.221790    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:03.649766 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:03.649723    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:04.776025 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:04.775989    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c27dca2f-2be7-4a83-bdc0-9df7082f0027" path="/var/lib/kubelet/pods/c27dca2f-2be7-4a83-bdc0-9df7082f0027/volumes"
Oct 30 23:03:05.159863 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:05.159724    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:05.880854 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:05.880814    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"]
Oct 30 23:03:05.884159 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:05.884126    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm" podUID="2ace3d4c-11b6-41ca-a276-37746b3791fa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:03:05.884510 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:05.884484    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995" podUID="e0a01454-5acd-417b-9e3c-62c826ef6558" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:03:05.989590 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:05.989554    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:06.535425 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:06.535396    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m"]
Oct 30 23:03:06.535622 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:06.535590    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m" podUID="0589bfca-27f5-4ba4-93af-9a0ffd07ffb6" containerName="maas-api" containerID="cri-o://340dcafb13d15c16c838d1f443419dc7b2beda283c0d992b4078cb67412cd1a0" gracePeriod=30
Oct 30 23:03:06.668033 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:06.668001    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:06.871072 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:06.871045    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m"
Oct 30 23:03:07.009250 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:06.996714    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-maas-api-tls\") pod \"0589bfca-27f5-4ba4-93af-9a0ffd07ffb6\" (UID: \"0589bfca-27f5-4ba4-93af-9a0ffd07ffb6\") "
Oct 30 23:03:07.009250 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:06.996784    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-kube-api-access-dh7qj\" (UniqueName: \"kubernetes.io/projected/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-kube-api-access-dh7qj\") pod \"0589bfca-27f5-4ba4-93af-9a0ffd07ffb6\" (UID: \"0589bfca-27f5-4ba4-93af-9a0ffd07ffb6\") "
Oct 30 23:03:07.009250 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:06.996863    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-metrics-certs\") pod \"0589bfca-27f5-4ba4-93af-9a0ffd07ffb6\" (UID: \"0589bfca-27f5-4ba4-93af-9a0ffd07ffb6\") "
Oct 30 23:03:07.009250 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:06.996905    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-openshift-service-ca\") pod \"0589bfca-27f5-4ba4-93af-9a0ffd07ffb6\" (UID: \"0589bfca-27f5-4ba4-93af-9a0ffd07ffb6\") "
Oct 30 23:03:07.009250 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:06.998330    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-openshift-service-ca" pod "0589bfca-27f5-4ba4-93af-9a0ffd07ffb6" (UID: "0589bfca-27f5-4ba4-93af-9a0ffd07ffb6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:07.038201 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:07.034867    4889 generic.go:356] "Generic (PLEG): container finished" podID="0589bfca-27f5-4ba4-93af-9a0ffd07ffb6" containerID="340dcafb13d15c16c838d1f443419dc7b2beda283c0d992b4078cb67412cd1a0" exitCode=0
Oct 30 23:03:07.038201 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:07.035039    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m" event={"ID":"0589bfca-27f5-4ba4-93af-9a0ffd07ffb6","Type":"ContainerDied","Data":"340dcafb13d15c16c838d1f443419dc7b2beda283c0d992b4078cb67412cd1a0"}
Oct 30 23:03:07.038201 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:07.035075    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m" event={"ID":"0589bfca-27f5-4ba4-93af-9a0ffd07ffb6","Type":"ContainerDied","Data":"2f1096eb0acf0b752a8cc15866a30f721f0c689db6aa2fb24aaa5da81526f1c6"}
Oct 30 23:03:07.038201 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:07.035098    4889 scope.go:122] "RemoveContainer" containerID="340dcafb13d15c16c838d1f443419dc7b2beda283c0d992b4078cb67412cd1a0"
Oct 30 23:03:07.038201 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:07.035204    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m"
Oct 30 23:03:07.046307 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:07.039903    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-metrics-certs" pod "0589bfca-27f5-4ba4-93af-9a0ffd07ffb6" (UID: "0589bfca-27f5-4ba4-93af-9a0ffd07ffb6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:07.046307 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:07.040371    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-kube-api-access-dh7qj" pod "0589bfca-27f5-4ba4-93af-9a0ffd07ffb6" (UID: "0589bfca-27f5-4ba4-93af-9a0ffd07ffb6"). InnerVolumeSpecName "kube-api-access-dh7qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:07.046307 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:07.041718    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-maas-api-tls" pod "0589bfca-27f5-4ba4-93af-9a0ffd07ffb6" (UID: "0589bfca-27f5-4ba4-93af-9a0ffd07ffb6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:07.075410 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:07.075386    4889 scope.go:122] "RemoveContainer" containerID="340dcafb13d15c16c838d1f443419dc7b2beda283c0d992b4078cb67412cd1a0"
Oct 30 23:03:07.075764 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:03:07.075743    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"340dcafb13d15c16c838d1f443419dc7b2beda283c0d992b4078cb67412cd1a0\": container with ID starting with 340dcafb13d15c16c838d1f443419dc7b2beda283c0d992b4078cb67412cd1a0 not found: ID does not exist" containerID="340dcafb13d15c16c838d1f443419dc7b2beda283c0d992b4078cb67412cd1a0"
Oct 30 23:03:07.075829 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:07.075777    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340dcafb13d15c16c838d1f443419dc7b2beda283c0d992b4078cb67412cd1a0"} err="failed to get container status \"340dcafb13d15c16c838d1f443419dc7b2beda283c0d992b4078cb67412cd1a0\": rpc error: code = NotFound desc = could not find container \"340dcafb13d15c16c838d1f443419dc7b2beda283c0d992b4078cb67412cd1a0\": container with ID starting with 340dcafb13d15c16c838d1f443419dc7b2beda283c0d992b4078cb67412cd1a0 not found: ID does not exist"
Oct 30 23:03:07.099219 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:07.099153    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:07.099219 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:07.099181    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh7qj\" (UniqueName: \"kubernetes.io/projected/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-kube-api-access-dh7qj\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:07.099219 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:07.099195    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:07.099219 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:07.099209    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:07.194206 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:07.194162    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:07.358453 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:07.358376    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:07.384772 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:07.384739    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m"]
Oct 30 23:03:07.398080 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:07.398053    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-d9f1c2-865f559dfb-sxc6m"]
Oct 30 23:03:07.533053 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:07.533017    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:07.875939 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:07.875908    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:08.775684 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:08.775651    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0589bfca-27f5-4ba4-93af-9a0ffd07ffb6" path="/var/lib/kubelet/pods/0589bfca-27f5-4ba4-93af-9a0ffd07ffb6/volumes"
Oct 30 23:03:09.326394 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:09.326357    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:10.302323 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:03:10.302284    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:03:10.713511 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:10.713416    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:10.895101 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:10.895064    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:11.071265 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:11.071214    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:11.298272 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:11.298238    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:13.051110 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:13.051070    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:14.467668 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:14.467636    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559"]
Oct 30 23:03:14.501070 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:14.501034    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg"]
Oct 30 23:03:14.708081 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:14.708049    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:14.913974 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:14.913943    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:15.086293 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:15.086264    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:15.272450 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:15.272414    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:15.758965 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:15.758921    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:15.884149 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:15.884109    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm" podUID="2ace3d4c-11b6-41ca-a276-37746b3791fa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:03:15.884734 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:15.884702    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995" podUID="e0a01454-5acd-417b-9e3c-62c826ef6558" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:03:16.802921 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:16.802885    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:18.531315 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:18.531267    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:18.949903 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:18.949876    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm"
Oct 30 23:03:18.990645 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:18.990620    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995"
Oct 30 23:03:19.081378 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.081343    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ace3d4c-11b6-41ca-a276-37746b3791fa-kube-api-access-t9wtz\" (UniqueName: \"kubernetes.io/projected/2ace3d4c-11b6-41ca-a276-37746b3791fa-kube-api-access-t9wtz\") pod \"2ace3d4c-11b6-41ca-a276-37746b3791fa\" (UID: \"2ace3d4c-11b6-41ca-a276-37746b3791fa\") "
Oct 30 23:03:19.081554 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.081427    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0a01454-5acd-417b-9e3c-62c826ef6558-kube-api-access-gqznk\" (UniqueName: \"kubernetes.io/projected/e0a01454-5acd-417b-9e3c-62c826ef6558-kube-api-access-gqznk\") pod \"e0a01454-5acd-417b-9e3c-62c826ef6558\" (UID: \"e0a01454-5acd-417b-9e3c-62c826ef6558\") "
Oct 30 23:03:19.081554 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.081471    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0a01454-5acd-417b-9e3c-62c826ef6558-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a01454-5acd-417b-9e3c-62c826ef6558-plugins-config-volume\") pod \"e0a01454-5acd-417b-9e3c-62c826ef6558\" (UID: \"e0a01454-5acd-417b-9e3c-62c826ef6558\") "
Oct 30 23:03:19.081688 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.081665    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ace3d4c-11b6-41ca-a276-37746b3791fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ace3d4c-11b6-41ca-a276-37746b3791fa-plugins-config-volume\") pod \"2ace3d4c-11b6-41ca-a276-37746b3791fa\" (UID: \"2ace3d4c-11b6-41ca-a276-37746b3791fa\") "
Oct 30 23:03:19.081891 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.081860    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0a01454-5acd-417b-9e3c-62c826ef6558-plugins-config-volume" pod "e0a01454-5acd-417b-9e3c-62c826ef6558" (UID: "e0a01454-5acd-417b-9e3c-62c826ef6558"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:19.082044 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.081994    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ace3d4c-11b6-41ca-a276-37746b3791fa-plugins-config-volume" pod "2ace3d4c-11b6-41ca-a276-37746b3791fa" (UID: "2ace3d4c-11b6-41ca-a276-37746b3791fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:19.082503 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.082481    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a01454-5acd-417b-9e3c-62c826ef6558-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:19.082557 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.082513    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ace3d4c-11b6-41ca-a276-37746b3791fa-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:19.083673 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.083651    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ace3d4c-11b6-41ca-a276-37746b3791fa-kube-api-access-t9wtz" pod "2ace3d4c-11b6-41ca-a276-37746b3791fa" (UID: "2ace3d4c-11b6-41ca-a276-37746b3791fa"). InnerVolumeSpecName "kube-api-access-t9wtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:19.083844 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.083816    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a01454-5acd-417b-9e3c-62c826ef6558-kube-api-access-gqznk" pod "e0a01454-5acd-417b-9e3c-62c826ef6558" (UID: "e0a01454-5acd-417b-9e3c-62c826ef6558"). InnerVolumeSpecName "kube-api-access-gqznk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:19.126913 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.126837    4889 generic.go:356] "Generic (PLEG): container finished" podID="e0a01454-5acd-417b-9e3c-62c826ef6558" containerID="fcb7050194b24fe2cfd5e1eda326c9a02a60a76014c202a89a1d592c48fcd3dd" exitCode=0
Oct 30 23:03:19.126913 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.126892    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995"
Oct 30 23:03:19.127108 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.126920    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995" event={"ID":"e0a01454-5acd-417b-9e3c-62c826ef6558","Type":"ContainerDied","Data":"fcb7050194b24fe2cfd5e1eda326c9a02a60a76014c202a89a1d592c48fcd3dd"}
Oct 30 23:03:19.127108 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.126975    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995" event={"ID":"e0a01454-5acd-417b-9e3c-62c826ef6558","Type":"ContainerDied","Data":"8808f44905c89c01e477c697d02a2842435024c1dc58d29788bbcfd2ff5a4e55"}
Oct 30 23:03:19.127108 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.126998    4889 scope.go:122] "RemoveContainer" containerID="fcb7050194b24fe2cfd5e1eda326c9a02a60a76014c202a89a1d592c48fcd3dd"
Oct 30 23:03:19.128771 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.128747    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm"
Oct 30 23:03:19.128888 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.128788    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm" event={"ID":"2ace3d4c-11b6-41ca-a276-37746b3791fa","Type":"ContainerDied","Data":"e42435c4b1d5891e13096494a67eb64caeb516f387614131dfd299be6d59616d"}
Oct 30 23:03:19.128888 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.128786    4889 generic.go:356] "Generic (PLEG): container finished" podID="2ace3d4c-11b6-41ca-a276-37746b3791fa" containerID="e42435c4b1d5891e13096494a67eb64caeb516f387614131dfd299be6d59616d" exitCode=0
Oct 30 23:03:19.128888 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.128874    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm" event={"ID":"2ace3d4c-11b6-41ca-a276-37746b3791fa","Type":"ContainerDied","Data":"80ed7084784c5c5c9f3ab78a1b70a107d3069d9a88f250efb066ad4c1af29ce4"}
Oct 30 23:03:19.138258 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.138218    4889 scope.go:122] "RemoveContainer" containerID="fcb7050194b24fe2cfd5e1eda326c9a02a60a76014c202a89a1d592c48fcd3dd"
Oct 30 23:03:19.138589 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:03:19.138568    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcb7050194b24fe2cfd5e1eda326c9a02a60a76014c202a89a1d592c48fcd3dd\": container with ID starting with fcb7050194b24fe2cfd5e1eda326c9a02a60a76014c202a89a1d592c48fcd3dd not found: ID does not exist" containerID="fcb7050194b24fe2cfd5e1eda326c9a02a60a76014c202a89a1d592c48fcd3dd"
Oct 30 23:03:19.138680 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.138601    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcb7050194b24fe2cfd5e1eda326c9a02a60a76014c202a89a1d592c48fcd3dd"} err="failed to get container status \"fcb7050194b24fe2cfd5e1eda326c9a02a60a76014c202a89a1d592c48fcd3dd\": rpc error: code = NotFound desc = could not find container \"fcb7050194b24fe2cfd5e1eda326c9a02a60a76014c202a89a1d592c48fcd3dd\": container with ID starting with fcb7050194b24fe2cfd5e1eda326c9a02a60a76014c202a89a1d592c48fcd3dd not found: ID does not exist"
Oct 30 23:03:19.138680 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.138625    4889 scope.go:122] "RemoveContainer" containerID="e42435c4b1d5891e13096494a67eb64caeb516f387614131dfd299be6d59616d"
Oct 30 23:03:19.148931 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.148911    4889 scope.go:122] "RemoveContainer" containerID="e42435c4b1d5891e13096494a67eb64caeb516f387614131dfd299be6d59616d"
Oct 30 23:03:19.149171 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:03:19.149154    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e42435c4b1d5891e13096494a67eb64caeb516f387614131dfd299be6d59616d\": container with ID starting with e42435c4b1d5891e13096494a67eb64caeb516f387614131dfd299be6d59616d not found: ID does not exist" containerID="e42435c4b1d5891e13096494a67eb64caeb516f387614131dfd299be6d59616d"
Oct 30 23:03:19.149266 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.149177    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e42435c4b1d5891e13096494a67eb64caeb516f387614131dfd299be6d59616d"} err="failed to get container status \"e42435c4b1d5891e13096494a67eb64caeb516f387614131dfd299be6d59616d\": rpc error: code = NotFound desc = could not find container \"e42435c4b1d5891e13096494a67eb64caeb516f387614131dfd299be6d59616d\": container with ID starting with e42435c4b1d5891e13096494a67eb64caeb516f387614131dfd299be6d59616d not found: ID does not exist"
Oct 30 23:03:19.183683 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.183657    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9wtz\" (UniqueName: \"kubernetes.io/projected/2ace3d4c-11b6-41ca-a276-37746b3791fa-kube-api-access-t9wtz\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:19.183683 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.183683    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqznk\" (UniqueName: \"kubernetes.io/projected/e0a01454-5acd-417b-9e3c-62c826ef6558-kube-api-access-gqznk\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:19.183857 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.183780    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm"]
Oct 30 23:03:19.187055 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.187033    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-510f80-c84b8b845-fpfjm"]
Oct 30 23:03:19.223596 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.223568    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995"]
Oct 30 23:03:19.229098 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.229076    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-510f80-54b6b8c75-9w995"]
Oct 30 23:03:19.468599 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.468493    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559" podUID="ba4dc809-cd8c-4957-ade8-9d9d434e20c9" containerName="payload-processing" containerID="cri-o://7a7dba4728cf1f3e0638bf2194ab4cc2d73f31fa4190bf816e9a761f17e698a8" gracePeriod=25
Oct 30 23:03:19.501841 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.501799    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg" podUID="6881b170-1a5e-4c71-8fc9-c45349847afd" containerName="payload-pre-processing" containerID="cri-o://38a2e8add39d24bff838dc985e64e4caf3ab2f0b08ed7111c17663068cdc3c90" gracePeriod=25
Oct 30 23:03:19.793950 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.793921    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559"
Oct 30 23:03:19.830895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.830856    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg"
Oct 30 23:03:19.892159 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.892122    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6881b170-1a5e-4c71-8fc9-c45349847afd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6881b170-1a5e-4c71-8fc9-c45349847afd-plugins-config-volume\") pod \"6881b170-1a5e-4c71-8fc9-c45349847afd\" (UID: \"6881b170-1a5e-4c71-8fc9-c45349847afd\") "
Oct 30 23:03:19.892394 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.892174    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba4dc809-cd8c-4957-ade8-9d9d434e20c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba4dc809-cd8c-4957-ade8-9d9d434e20c9-plugins-config-volume\") pod \"ba4dc809-cd8c-4957-ade8-9d9d434e20c9\" (UID: \"ba4dc809-cd8c-4957-ade8-9d9d434e20c9\") "
Oct 30 23:03:19.892394 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.892290    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba4dc809-cd8c-4957-ade8-9d9d434e20c9-kube-api-access-jjzbg\" (UniqueName: \"kubernetes.io/projected/ba4dc809-cd8c-4957-ade8-9d9d434e20c9-kube-api-access-jjzbg\") pod \"ba4dc809-cd8c-4957-ade8-9d9d434e20c9\" (UID: \"ba4dc809-cd8c-4957-ade8-9d9d434e20c9\") "
Oct 30 23:03:19.892394 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.892330    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6881b170-1a5e-4c71-8fc9-c45349847afd-kube-api-access-ns5pf\" (UniqueName: \"kubernetes.io/projected/6881b170-1a5e-4c71-8fc9-c45349847afd-kube-api-access-ns5pf\") pod \"6881b170-1a5e-4c71-8fc9-c45349847afd\" (UID: \"6881b170-1a5e-4c71-8fc9-c45349847afd\") "
Oct 30 23:03:19.892650 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.892620    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6881b170-1a5e-4c71-8fc9-c45349847afd-plugins-config-volume" pod "6881b170-1a5e-4c71-8fc9-c45349847afd" (UID: "6881b170-1a5e-4c71-8fc9-c45349847afd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:19.893453 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.892912    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba4dc809-cd8c-4957-ade8-9d9d434e20c9-plugins-config-volume" pod "ba4dc809-cd8c-4957-ade8-9d9d434e20c9" (UID: "ba4dc809-cd8c-4957-ade8-9d9d434e20c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:19.893453 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.893422    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6881b170-1a5e-4c71-8fc9-c45349847afd-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:19.893453 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.893445    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba4dc809-cd8c-4957-ade8-9d9d434e20c9-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:19.895695 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.895662    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba4dc809-cd8c-4957-ade8-9d9d434e20c9-kube-api-access-jjzbg" pod "ba4dc809-cd8c-4957-ade8-9d9d434e20c9" (UID: "ba4dc809-cd8c-4957-ade8-9d9d434e20c9"). InnerVolumeSpecName "kube-api-access-jjzbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:19.897869 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.897805    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6881b170-1a5e-4c71-8fc9-c45349847afd-kube-api-access-ns5pf" pod "6881b170-1a5e-4c71-8fc9-c45349847afd" (UID: "6881b170-1a5e-4c71-8fc9-c45349847afd"). InnerVolumeSpecName "kube-api-access-ns5pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:19.957171 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.957131    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:19.994952 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.994916    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjzbg\" (UniqueName: \"kubernetes.io/projected/ba4dc809-cd8c-4957-ade8-9d9d434e20c9-kube-api-access-jjzbg\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:19.994952 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:19.994950    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns5pf\" (UniqueName: \"kubernetes.io/projected/6881b170-1a5e-4c71-8fc9-c45349847afd-kube-api-access-ns5pf\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:20.137257 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.137201    4889 generic.go:356] "Generic (PLEG): container finished" podID="ba4dc809-cd8c-4957-ade8-9d9d434e20c9" containerID="7a7dba4728cf1f3e0638bf2194ab4cc2d73f31fa4190bf816e9a761f17e698a8" exitCode=0
Oct 30 23:03:20.137470 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.137288    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559"
Oct 30 23:03:20.137470 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.137294    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559" event={"ID":"ba4dc809-cd8c-4957-ade8-9d9d434e20c9","Type":"ContainerDied","Data":"7a7dba4728cf1f3e0638bf2194ab4cc2d73f31fa4190bf816e9a761f17e698a8"}
Oct 30 23:03:20.137470 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.137352    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559" event={"ID":"ba4dc809-cd8c-4957-ade8-9d9d434e20c9","Type":"ContainerDied","Data":"7701946ca7b18faf4938f787270b36dc63c4fd29c61249b371a8039ba5c49fc8"}
Oct 30 23:03:20.137470 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.137383    4889 scope.go:122] "RemoveContainer" containerID="7a7dba4728cf1f3e0638bf2194ab4cc2d73f31fa4190bf816e9a761f17e698a8"
Oct 30 23:03:20.138907 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.138885    4889 generic.go:356] "Generic (PLEG): container finished" podID="6881b170-1a5e-4c71-8fc9-c45349847afd" containerID="38a2e8add39d24bff838dc985e64e4caf3ab2f0b08ed7111c17663068cdc3c90" exitCode=0
Oct 30 23:03:20.139017 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.138930    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg"
Oct 30 23:03:20.139017 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.138944    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg" event={"ID":"6881b170-1a5e-4c71-8fc9-c45349847afd","Type":"ContainerDied","Data":"38a2e8add39d24bff838dc985e64e4caf3ab2f0b08ed7111c17663068cdc3c90"}
Oct 30 23:03:20.139017 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.138992    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg" event={"ID":"6881b170-1a5e-4c71-8fc9-c45349847afd","Type":"ContainerDied","Data":"af962a1d3c38f8cfa68e4829115cbe0c8aff779bb2f0f4b6f26272f9c512a6eb"}
Oct 30 23:03:20.150706 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.150683    4889 scope.go:122] "RemoveContainer" containerID="7a7dba4728cf1f3e0638bf2194ab4cc2d73f31fa4190bf816e9a761f17e698a8"
Oct 30 23:03:20.151059 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:03:20.151036    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a7dba4728cf1f3e0638bf2194ab4cc2d73f31fa4190bf816e9a761f17e698a8\": container with ID starting with 7a7dba4728cf1f3e0638bf2194ab4cc2d73f31fa4190bf816e9a761f17e698a8 not found: ID does not exist" containerID="7a7dba4728cf1f3e0638bf2194ab4cc2d73f31fa4190bf816e9a761f17e698a8"
Oct 30 23:03:20.151175 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.151067    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a7dba4728cf1f3e0638bf2194ab4cc2d73f31fa4190bf816e9a761f17e698a8"} err="failed to get container status \"7a7dba4728cf1f3e0638bf2194ab4cc2d73f31fa4190bf816e9a761f17e698a8\": rpc error: code = NotFound desc = could not find container \"7a7dba4728cf1f3e0638bf2194ab4cc2d73f31fa4190bf816e9a761f17e698a8\": container with ID starting with 7a7dba4728cf1f3e0638bf2194ab4cc2d73f31fa4190bf816e9a761f17e698a8 not found: ID does not exist"
Oct 30 23:03:20.151175 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.151087    4889 scope.go:122] "RemoveContainer" containerID="38a2e8add39d24bff838dc985e64e4caf3ab2f0b08ed7111c17663068cdc3c90"
Oct 30 23:03:20.155196 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.155158    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:20.163480 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.163453    4889 scope.go:122] "RemoveContainer" containerID="38a2e8add39d24bff838dc985e64e4caf3ab2f0b08ed7111c17663068cdc3c90"
Oct 30 23:03:20.163857 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:03:20.163834    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38a2e8add39d24bff838dc985e64e4caf3ab2f0b08ed7111c17663068cdc3c90\": container with ID starting with 38a2e8add39d24bff838dc985e64e4caf3ab2f0b08ed7111c17663068cdc3c90 not found: ID does not exist" containerID="38a2e8add39d24bff838dc985e64e4caf3ab2f0b08ed7111c17663068cdc3c90"
Oct 30 23:03:20.163926 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.163868    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38a2e8add39d24bff838dc985e64e4caf3ab2f0b08ed7111c17663068cdc3c90"} err="failed to get container status \"38a2e8add39d24bff838dc985e64e4caf3ab2f0b08ed7111c17663068cdc3c90\": rpc error: code = NotFound desc = could not find container \"38a2e8add39d24bff838dc985e64e4caf3ab2f0b08ed7111c17663068cdc3c90\": container with ID starting with 38a2e8add39d24bff838dc985e64e4caf3ab2f0b08ed7111c17663068cdc3c90 not found: ID does not exist"
Oct 30 23:03:20.196553 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.196520    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559"]
Oct 30 23:03:20.202952 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.202913    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bff950-746d89d77f-w4559"]
Oct 30 23:03:20.240742 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.240710    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg"]
Oct 30 23:03:20.253511 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.253437    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-d477c7687-vm8sg"]
Oct 30 23:03:20.329059 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.329018    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:20.522302 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.522199    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:20.715346 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.715309    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:20.775792 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.775713    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ace3d4c-11b6-41ca-a276-37746b3791fa" path="/var/lib/kubelet/pods/2ace3d4c-11b6-41ca-a276-37746b3791fa/volumes"
Oct 30 23:03:20.776172 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.776159    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6881b170-1a5e-4c71-8fc9-c45349847afd" path="/var/lib/kubelet/pods/6881b170-1a5e-4c71-8fc9-c45349847afd/volumes"
Oct 30 23:03:20.776654 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.776637    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba4dc809-cd8c-4957-ade8-9d9d434e20c9" path="/var/lib/kubelet/pods/ba4dc809-cd8c-4957-ade8-9d9d434e20c9/volumes"
Oct 30 23:03:20.777004 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.776993    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0a01454-5acd-417b-9e3c-62c826ef6558" path="/var/lib/kubelet/pods/e0a01454-5acd-417b-9e3c-62c826ef6558/volumes"
Oct 30 23:03:20.878380 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.878349    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:20.908638 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:20.908602    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6" podUID="1cd0f933-b208-424d-a9c1-04f8c6044054" containerName="main" containerID="cri-o://df760719ade8961ccddef5d9380fd5a3e91c64a7215562318e68bba5457d7e05" gracePeriod=45
Oct 30 23:03:21.151705 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:21.151666    4889 generic.go:356] "Generic (PLEG): container finished" podID="1cd0f933-b208-424d-a9c1-04f8c6044054" containerID="df760719ade8961ccddef5d9380fd5a3e91c64a7215562318e68bba5457d7e05" exitCode=0
Oct 30 23:03:21.151865 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:21.151740    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6" event={"ID":"1cd0f933-b208-424d-a9c1-04f8c6044054","Type":"ContainerDied","Data":"df760719ade8961ccddef5d9380fd5a3e91c64a7215562318e68bba5457d7e05"}
Oct 30 23:03:21.166473 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:21.166437    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:21.195166 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:21.195133    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"
Oct 30 23:03:21.314609 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:21.314577    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cd0f933-b208-424d-a9c1-04f8c6044054-tls-certs\" (UniqueName: \"kubernetes.io/secret/1cd0f933-b208-424d-a9c1-04f8c6044054-tls-certs\") pod \"1cd0f933-b208-424d-a9c1-04f8c6044054\" (UID: \"1cd0f933-b208-424d-a9c1-04f8c6044054\") "
Oct 30 23:03:21.314860 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:21.314660    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cd0f933-b208-424d-a9c1-04f8c6044054-kube-api-access-8hr7k\" (UniqueName: \"kubernetes.io/projected/1cd0f933-b208-424d-a9c1-04f8c6044054-kube-api-access-8hr7k\") pod \"1cd0f933-b208-424d-a9c1-04f8c6044054\" (UID: \"1cd0f933-b208-424d-a9c1-04f8c6044054\") "
Oct 30 23:03:21.314860 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:21.314691    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-dshm\") pod \"1cd0f933-b208-424d-a9c1-04f8c6044054\" (UID: \"1cd0f933-b208-424d-a9c1-04f8c6044054\") "
Oct 30 23:03:21.314860 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:21.314726    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-model-cache\") pod \"1cd0f933-b208-424d-a9c1-04f8c6044054\" (UID: \"1cd0f933-b208-424d-a9c1-04f8c6044054\") "
Oct 30 23:03:21.314860 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:21.314844    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-tmp-dir\") pod \"1cd0f933-b208-424d-a9c1-04f8c6044054\" (UID: \"1cd0f933-b208-424d-a9c1-04f8c6044054\") "
Oct 30 23:03:21.315111 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:21.315012    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-home\" (UniqueName: \"kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-home\") pod \"1cd0f933-b208-424d-a9c1-04f8c6044054\" (UID: \"1cd0f933-b208-424d-a9c1-04f8c6044054\") "
Oct 30 23:03:21.315111 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:21.315075    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-model-cache" pod "1cd0f933-b208-424d-a9c1-04f8c6044054" (UID: "1cd0f933-b208-424d-a9c1-04f8c6044054"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:21.315111 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:21.315084    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-tmp-dir" pod "1cd0f933-b208-424d-a9c1-04f8c6044054" (UID: "1cd0f933-b208-424d-a9c1-04f8c6044054"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:21.315571 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:21.315549    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:21.315571 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:21.315570    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:21.315807 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:21.315783    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-home" pod "1cd0f933-b208-424d-a9c1-04f8c6044054" (UID: "1cd0f933-b208-424d-a9c1-04f8c6044054"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:21.317192 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:21.317166    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd0f933-b208-424d-a9c1-04f8c6044054-kube-api-access-8hr7k" pod "1cd0f933-b208-424d-a9c1-04f8c6044054" (UID: "1cd0f933-b208-424d-a9c1-04f8c6044054"). InnerVolumeSpecName "kube-api-access-8hr7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:21.317399 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:21.317373    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cd0f933-b208-424d-a9c1-04f8c6044054-tls-certs" pod "1cd0f933-b208-424d-a9c1-04f8c6044054" (UID: "1cd0f933-b208-424d-a9c1-04f8c6044054"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:21.318487 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:21.318469    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-dshm" pod "1cd0f933-b208-424d-a9c1-04f8c6044054" (UID: "1cd0f933-b208-424d-a9c1-04f8c6044054"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:21.416913 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:21.416823    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:21.416913 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:21.416858    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cd0f933-b208-424d-a9c1-04f8c6044054-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:21.416913 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:21.416870    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hr7k\" (UniqueName: \"kubernetes.io/projected/1cd0f933-b208-424d-a9c1-04f8c6044054-kube-api-access-8hr7k\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:21.416913 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:21.416880    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cd0f933-b208-424d-a9c1-04f8c6044054-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:21.951476 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:21.951437    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"]
Oct 30 23:03:22.045970 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:22.045915    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:22.165316 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:22.165285    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"
Oct 30 23:03:22.165316 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:22.165293    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6" event={"ID":"1cd0f933-b208-424d-a9c1-04f8c6044054","Type":"ContainerDied","Data":"30fce40e1fe792553a8db1d6c3857df1924886d34b5ac68df0bb4e5abf55dc6b"}
Oct 30 23:03:22.165614 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:22.165357    4889 scope.go:122] "RemoveContainer" containerID="df760719ade8961ccddef5d9380fd5a3e91c64a7215562318e68bba5457d7e05"
Oct 30 23:03:22.222303 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:22.222136    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"]
Oct 30 23:03:22.226128 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:22.226100    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-bed544/sub-test-model-bed544-kserve-746c6658fc-qtgw6"]
Oct 30 23:03:22.260886 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:22.260850    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:22.776152 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:22.776110    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cd0f933-b208-424d-a9c1-04f8c6044054" path="/var/lib/kubelet/pods/1cd0f933-b208-424d-a9c1-04f8c6044054/volumes"
Oct 30 23:03:22.786654 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:22.786622    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:23.012023 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:23.011990    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:23.056805 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:23.056728    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx"]
Oct 30 23:03:23.183696 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:23.183662    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:23.366996 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:23.366902    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:23.534285 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:23.534250    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:25.435038 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:25.434986    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:26.401544 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:26.401498    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6"]
Oct 30 23:03:26.434522 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:26.434485    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc"]
Oct 30 23:03:26.598800 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:26.598765    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:26.767691 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:26.767648    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:27.240071 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:27.239975    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:27.820952 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:27.820917    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:27.992391 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:27.992357    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:28.057458 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:28.057417    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx" podUID="cc0df870-2a4a-404b-9678-bdc054dce3e3" containerName="payload-pre-processing" containerID="cri-o://dc70155ffafe9e975a8ca507d202284df36f0ae94460753a5267febbe8c22293" gracePeriod=25
Oct 30 23:03:28.086893 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:28.086812    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"]
Oct 30 23:03:28.087052 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:28.087030    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2" podUID="1aa5f50b-7768-46fd-9ce4-76c47268bce2" containerName="istio-proxy" containerID="cri-o://df043c0e8586f7d3805171f6196ccf639a8fb53fd6c54891b96fb103e1b34bbd" gracePeriod=30
Oct 30 23:03:28.180660 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:28.180630    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:29.139249 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:29.131044    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:30.291671 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:30.291633    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:30.456582 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:30.456547    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:30.547906 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:30.547819    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"]
Oct 30 23:03:30.548360 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:30.548325    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf" podUID="d1cef895-22cd-4599-8239-d2ebf69cd234" containerName="istio-proxy" containerID="cri-o://7fa6f3f6db1f4c704a6ae567eff11c13eb96b27adc7cfb26a8216f5aa93cde38" gracePeriod=30
Oct 30 23:03:30.648252 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:30.646253    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:31.094185 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:31.094144    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:31.269599 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:31.269556    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:31.402876 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:31.402779    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6" podUID="b812174d-906c-487b-91a9-7a1620aca44e" containerName="payload-processing" containerID="cri-o://632093beb73e141eed51a24f65cd85fb6454fb25f32d92a7100dee5e129bbbb0" gracePeriod=25
Oct 30 23:03:31.435087 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:31.435047    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc" podUID="2e0a2179-dbbd-49c3-b6af-90aec117a44c" containerName="payload-pre-processing" containerID="cri-o://dcf269360f829facc20b88c5d510c5292f50e8a0690a69f665911fb548b1f89e" gracePeriod=25
Oct 30 23:03:32.688908 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:32.688860    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv"]
Oct 30 23:03:32.723775 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:32.723743    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv"]
Oct 30 23:03:32.874674 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:32.874642    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:33.050407 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.050366    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:33.272629 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.272199    4889 generic.go:356] "Generic (PLEG): container finished" podID="1aa5f50b-7768-46fd-9ce4-76c47268bce2" containerID="df043c0e8586f7d3805171f6196ccf639a8fb53fd6c54891b96fb103e1b34bbd" exitCode=0
Oct 30 23:03:33.272629 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.272431    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2" event={"ID":"1aa5f50b-7768-46fd-9ce4-76c47268bce2","Type":"ContainerDied","Data":"df043c0e8586f7d3805171f6196ccf639a8fb53fd6c54891b96fb103e1b34bbd"}
Oct 30 23:03:33.272629 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.272497    4889 scope.go:122] "RemoveContainer" containerID="60c45600e6986249c58dfc88800b63f8a6711b0db3111f5f480ece96a3b8be40"
Oct 30 23:03:33.410972 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.410948    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 23:03:33.517332 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.517302    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-data\") pod \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") "
Oct 30 23:03:33.517507 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.517419    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-envoy\") pod \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") "
Oct 30 23:03:33.517507 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.517444    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-credential-socket\") pod \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") "
Oct 30 23:03:33.517592 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.517520    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istiod-ca-cert\") pod \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") "
Oct 30 23:03:33.517592 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.517540    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aa5f50b-7768-46fd-9ce4-76c47268bce2-kube-api-access-vdnbh\" (UniqueName: \"kubernetes.io/projected/1aa5f50b-7768-46fd-9ce4-76c47268bce2-kube-api-access-vdnbh\") pod \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") "
Oct 30 23:03:33.517592 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.517566    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-data" pod "1aa5f50b-7768-46fd-9ce4-76c47268bce2" (UID: "1aa5f50b-7768-46fd-9ce4-76c47268bce2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:33.517592 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.517583    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-token\" (UniqueName: \"kubernetes.io/projected/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-token\") pod \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") "
Oct 30 23:03:33.517803 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.517620    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-podinfo\") pod \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") "
Oct 30 23:03:33.517803 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.517675    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-workload-socket\") pod \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") "
Oct 30 23:03:33.517803 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.517708    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-workload-certs\") pod \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\" (UID: \"1aa5f50b-7768-46fd-9ce4-76c47268bce2\") "
Oct 30 23:03:33.517986 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.517826    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-credential-socket" pod "1aa5f50b-7768-46fd-9ce4-76c47268bce2" (UID: "1aa5f50b-7768-46fd-9ce4-76c47268bce2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:33.518063 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.517935    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istiod-ca-cert" pod "1aa5f50b-7768-46fd-9ce4-76c47268bce2" (UID: "1aa5f50b-7768-46fd-9ce4-76c47268bce2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:33.518063 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.518023    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-workload-socket" pod "1aa5f50b-7768-46fd-9ce4-76c47268bce2" (UID: "1aa5f50b-7768-46fd-9ce4-76c47268bce2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:33.518259 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.518200    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-workload-certs" pod "1aa5f50b-7768-46fd-9ce4-76c47268bce2" (UID: "1aa5f50b-7768-46fd-9ce4-76c47268bce2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:33.518534 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.518513    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:33.518614 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.518540    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:33.518614 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.518555    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:33.518614 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.518568    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:33.518614 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.518582    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:33.519972 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.519951    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-envoy" pod "1aa5f50b-7768-46fd-9ce4-76c47268bce2" (UID: "1aa5f50b-7768-46fd-9ce4-76c47268bce2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:33.520101 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.520009    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa5f50b-7768-46fd-9ce4-76c47268bce2-kube-api-access-vdnbh" pod "1aa5f50b-7768-46fd-9ce4-76c47268bce2" (UID: "1aa5f50b-7768-46fd-9ce4-76c47268bce2"). InnerVolumeSpecName "kube-api-access-vdnbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:33.520101 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.520034    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-token" pod "1aa5f50b-7768-46fd-9ce4-76c47268bce2" (UID: "1aa5f50b-7768-46fd-9ce4-76c47268bce2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:33.520387 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.520370    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-podinfo" pod "1aa5f50b-7768-46fd-9ce4-76c47268bce2" (UID: "1aa5f50b-7768-46fd-9ce4-76c47268bce2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:03:33.620306 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.620218    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdnbh\" (UniqueName: \"kubernetes.io/projected/1aa5f50b-7768-46fd-9ce4-76c47268bce2-kube-api-access-vdnbh\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:33.620306 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.620260    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:33.620306 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.620270    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:33.620306 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:33.620278    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa5f50b-7768-46fd-9ce4-76c47268bce2-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:34.280595 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:34.280563    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"
Oct 30 23:03:34.281140 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:34.280596    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2" event={"ID":"1aa5f50b-7768-46fd-9ce4-76c47268bce2","Type":"ContainerDied","Data":"f67d0e3132f28452bba8077e9aad6f603c8e9bebb113e18e435877f22259efa0"}
Oct 30 23:03:34.281140 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:34.280672    4889 scope.go:122] "RemoveContainer" containerID="df043c0e8586f7d3805171f6196ccf639a8fb53fd6c54891b96fb103e1b34bbd"
Oct 30 23:03:34.342575 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:34.342510    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"]
Oct 30 23:03:34.355910 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:34.355871    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-76d54a-openshift-default-6bcffbc9bc-8d4d2"]
Oct 30 23:03:34.622470 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:34.622372    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:34.775473 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:34.775443    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aa5f50b-7768-46fd-9ce4-76c47268bce2" path="/var/lib/kubelet/pods/1aa5f50b-7768-46fd-9ce4-76c47268bce2/volumes"
Oct 30 23:03:34.826078 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:34.826040    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx" podUID="cc0df870-2a4a-404b-9678-bdc054dce3e3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:03:35.842204 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:35.842175    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 23:03:35.948969 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:35.948933    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-envoy\") pod \"d1cef895-22cd-4599-8239-d2ebf69cd234\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") "
Oct 30 23:03:35.949290 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:35.949268    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-podinfo\") pod \"d1cef895-22cd-4599-8239-d2ebf69cd234\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") "
Oct 30 23:03:35.949468 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:35.949451    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-token\" (UniqueName: \"kubernetes.io/projected/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-token\") pod \"d1cef895-22cd-4599-8239-d2ebf69cd234\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") "
Oct 30 23:03:35.949619 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:35.949601    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1cef895-22cd-4599-8239-d2ebf69cd234-kube-api-access-8zdtt\" (UniqueName: \"kubernetes.io/projected/d1cef895-22cd-4599-8239-d2ebf69cd234-kube-api-access-8zdtt\") pod \"d1cef895-22cd-4599-8239-d2ebf69cd234\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") "
Oct 30 23:03:35.949819 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:35.949805    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-credential-socket\") pod \"d1cef895-22cd-4599-8239-d2ebf69cd234\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") "
Oct 30 23:03:35.950000 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:35.949986    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1cef895-22cd-4599-8239-d2ebf69cd234-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1cef895-22cd-4599-8239-d2ebf69cd234-istiod-ca-cert\") pod \"d1cef895-22cd-4599-8239-d2ebf69cd234\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") "
Oct 30 23:03:35.950135 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:35.950122    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-workload-certs\") pod \"d1cef895-22cd-4599-8239-d2ebf69cd234\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") "
Oct 30 23:03:35.950320 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:35.950305    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-data\") pod \"d1cef895-22cd-4599-8239-d2ebf69cd234\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") "
Oct 30 23:03:35.950454 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:35.950440    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-workload-socket\") pod \"d1cef895-22cd-4599-8239-d2ebf69cd234\" (UID: \"d1cef895-22cd-4599-8239-d2ebf69cd234\") "
Oct 30 23:03:35.953297 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:35.953271    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-workload-socket" pod "d1cef895-22cd-4599-8239-d2ebf69cd234" (UID: "d1cef895-22cd-4599-8239-d2ebf69cd234"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:35.953410 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:35.953319    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr"]
Oct 30 23:03:35.953582 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:35.953558    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr" podUID="a3ff065c-81fb-4272-b8e1-0b8213ecbd19" containerName="maas-api" containerID="cri-o://74574b111d513140f217dba86f11e46e06a638aeccdfd084bf0ae0fd9c3d872d" gracePeriod=30
Oct 30 23:03:35.958997 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:35.958971    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-envoy" pod "d1cef895-22cd-4599-8239-d2ebf69cd234" (UID: "d1cef895-22cd-4599-8239-d2ebf69cd234"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:35.959482 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:35.959453    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-workload-certs" pod "d1cef895-22cd-4599-8239-d2ebf69cd234" (UID: "d1cef895-22cd-4599-8239-d2ebf69cd234"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:35.959663 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:35.959643    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-data" pod "d1cef895-22cd-4599-8239-d2ebf69cd234" (UID: "d1cef895-22cd-4599-8239-d2ebf69cd234"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:35.960403 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:35.960207    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1cef895-22cd-4599-8239-d2ebf69cd234-istiod-ca-cert" pod "d1cef895-22cd-4599-8239-d2ebf69cd234" (UID: "d1cef895-22cd-4599-8239-d2ebf69cd234"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:35.962423 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:35.962397    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-credential-socket" pod "d1cef895-22cd-4599-8239-d2ebf69cd234" (UID: "d1cef895-22cd-4599-8239-d2ebf69cd234"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:35.970792 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:35.970762    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-podinfo" pod "d1cef895-22cd-4599-8239-d2ebf69cd234" (UID: "d1cef895-22cd-4599-8239-d2ebf69cd234"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:03:35.971148 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:35.971122    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-token" pod "d1cef895-22cd-4599-8239-d2ebf69cd234" (UID: "d1cef895-22cd-4599-8239-d2ebf69cd234"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:35.972956 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:35.972930    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1cef895-22cd-4599-8239-d2ebf69cd234-kube-api-access-8zdtt" pod "d1cef895-22cd-4599-8239-d2ebf69cd234" (UID: "d1cef895-22cd-4599-8239-d2ebf69cd234"). InnerVolumeSpecName "kube-api-access-8zdtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:36.017560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.017528    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"]
Oct 30 23:03:36.052942 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.052906    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.052942 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.052936    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1cef895-22cd-4599-8239-d2ebf69cd234-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.052942 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.052947    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.053188 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.052955    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.053188 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.052969    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.053188 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.052981    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.053188 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.052988    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.053188 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.053000    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1cef895-22cd-4599-8239-d2ebf69cd234-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.053188 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.053013    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zdtt\" (UniqueName: \"kubernetes.io/projected/d1cef895-22cd-4599-8239-d2ebf69cd234-kube-api-access-8zdtt\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.095894 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.095850    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:36.287084 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.287050    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr"
Oct 30 23:03:36.297946 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.297912    4889 generic.go:356] "Generic (PLEG): container finished" podID="a3ff065c-81fb-4272-b8e1-0b8213ecbd19" containerID="74574b111d513140f217dba86f11e46e06a638aeccdfd084bf0ae0fd9c3d872d" exitCode=0
Oct 30 23:03:36.298130 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.297970    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr"
Oct 30 23:03:36.298130 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.298006    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr" event={"ID":"a3ff065c-81fb-4272-b8e1-0b8213ecbd19","Type":"ContainerDied","Data":"74574b111d513140f217dba86f11e46e06a638aeccdfd084bf0ae0fd9c3d872d"}
Oct 30 23:03:36.298130 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.298071    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr" event={"ID":"a3ff065c-81fb-4272-b8e1-0b8213ecbd19","Type":"ContainerDied","Data":"b4b7af42ffa74272055441b8f455e5e3a4b0f78fb870686e1dab840ba55d9d20"}
Oct 30 23:03:36.298130 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.298096    4889 scope.go:122] "RemoveContainer" containerID="74574b111d513140f217dba86f11e46e06a638aeccdfd084bf0ae0fd9c3d872d"
Oct 30 23:03:36.299761 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.299728    4889 generic.go:356] "Generic (PLEG): container finished" podID="d1cef895-22cd-4599-8239-d2ebf69cd234" containerID="7fa6f3f6db1f4c704a6ae567eff11c13eb96b27adc7cfb26a8216f5aa93cde38" exitCode=0
Oct 30 23:03:36.299861 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.299771    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"
Oct 30 23:03:36.299861 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.299810    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf" event={"ID":"d1cef895-22cd-4599-8239-d2ebf69cd234","Type":"ContainerDied","Data":"7fa6f3f6db1f4c704a6ae567eff11c13eb96b27adc7cfb26a8216f5aa93cde38"}
Oct 30 23:03:36.299955 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.299861    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf" event={"ID":"d1cef895-22cd-4599-8239-d2ebf69cd234","Type":"ContainerDied","Data":"a20252d458c309611ec5f90a2be55fe98b1b49484bcf21030846f64729a3e933"}
Oct 30 23:03:36.310120 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.309833    4889 scope.go:122] "RemoveContainer" containerID="74574b111d513140f217dba86f11e46e06a638aeccdfd084bf0ae0fd9c3d872d"
Oct 30 23:03:36.310327 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:03:36.310154    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74574b111d513140f217dba86f11e46e06a638aeccdfd084bf0ae0fd9c3d872d\": container with ID starting with 74574b111d513140f217dba86f11e46e06a638aeccdfd084bf0ae0fd9c3d872d not found: ID does not exist" containerID="74574b111d513140f217dba86f11e46e06a638aeccdfd084bf0ae0fd9c3d872d"
Oct 30 23:03:36.310327 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.310181    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74574b111d513140f217dba86f11e46e06a638aeccdfd084bf0ae0fd9c3d872d"} err="failed to get container status \"74574b111d513140f217dba86f11e46e06a638aeccdfd084bf0ae0fd9c3d872d\": rpc error: code = NotFound desc = could not find container \"74574b111d513140f217dba86f11e46e06a638aeccdfd084bf0ae0fd9c3d872d\": container with ID starting with 74574b111d513140f217dba86f11e46e06a638aeccdfd084bf0ae0fd9c3d872d not found: ID does not exist"
Oct 30 23:03:36.310327 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.310200    4889 scope.go:122] "RemoveContainer" containerID="7fa6f3f6db1f4c704a6ae567eff11c13eb96b27adc7cfb26a8216f5aa93cde38"
Oct 30 23:03:36.321579 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.321556    4889 scope.go:122] "RemoveContainer" containerID="7fa6f3f6db1f4c704a6ae567eff11c13eb96b27adc7cfb26a8216f5aa93cde38"
Oct 30 23:03:36.321933 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:03:36.321908    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fa6f3f6db1f4c704a6ae567eff11c13eb96b27adc7cfb26a8216f5aa93cde38\": container with ID starting with 7fa6f3f6db1f4c704a6ae567eff11c13eb96b27adc7cfb26a8216f5aa93cde38 not found: ID does not exist" containerID="7fa6f3f6db1f4c704a6ae567eff11c13eb96b27adc7cfb26a8216f5aa93cde38"
Oct 30 23:03:36.321991 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.321945    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fa6f3f6db1f4c704a6ae567eff11c13eb96b27adc7cfb26a8216f5aa93cde38"} err="failed to get container status \"7fa6f3f6db1f4c704a6ae567eff11c13eb96b27adc7cfb26a8216f5aa93cde38\": rpc error: code = NotFound desc = could not find container \"7fa6f3f6db1f4c704a6ae567eff11c13eb96b27adc7cfb26a8216f5aa93cde38\": container with ID starting with 7fa6f3f6db1f4c704a6ae567eff11c13eb96b27adc7cfb26a8216f5aa93cde38 not found: ID does not exist"
Oct 30 23:03:36.344975 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.344814    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:36.357713 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.357686    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-maas-api-tls\") pod \"a3ff065c-81fb-4272-b8e1-0b8213ecbd19\" (UID: \"a3ff065c-81fb-4272-b8e1-0b8213ecbd19\") "
Oct 30 23:03:36.357875 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.357748    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-metrics-certs\") pod \"a3ff065c-81fb-4272-b8e1-0b8213ecbd19\" (UID: \"a3ff065c-81fb-4272-b8e1-0b8213ecbd19\") "
Oct 30 23:03:36.357963 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.357894    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-kube-api-access-n7s9v\" (UniqueName: \"kubernetes.io/projected/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-kube-api-access-n7s9v\") pod \"a3ff065c-81fb-4272-b8e1-0b8213ecbd19\" (UID: \"a3ff065c-81fb-4272-b8e1-0b8213ecbd19\") "
Oct 30 23:03:36.357963 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.357953    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-openshift-service-ca\") pod \"a3ff065c-81fb-4272-b8e1-0b8213ecbd19\" (UID: \"a3ff065c-81fb-4272-b8e1-0b8213ecbd19\") "
Oct 30 23:03:36.358371 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.358348    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-openshift-service-ca" pod "a3ff065c-81fb-4272-b8e1-0b8213ecbd19" (UID: "a3ff065c-81fb-4272-b8e1-0b8213ecbd19"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:36.358494 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.358478    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.360945 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.360916    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-kube-api-access-n7s9v" pod "a3ff065c-81fb-4272-b8e1-0b8213ecbd19" (UID: "a3ff065c-81fb-4272-b8e1-0b8213ecbd19"). InnerVolumeSpecName "kube-api-access-n7s9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:36.361587 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.361555    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-metrics-certs" pod "a3ff065c-81fb-4272-b8e1-0b8213ecbd19" (UID: "a3ff065c-81fb-4272-b8e1-0b8213ecbd19"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:36.363369 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.363340    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-maas-api-tls" pod "a3ff065c-81fb-4272-b8e1-0b8213ecbd19" (UID: "a3ff065c-81fb-4272-b8e1-0b8213ecbd19"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:36.391299 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.391105    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"]
Oct 30 23:03:36.396743 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.396715    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-4qscf"]
Oct 30 23:03:36.459470 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.459432    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7s9v\" (UniqueName: \"kubernetes.io/projected/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-kube-api-access-n7s9v\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.459470 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.459462    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.459470 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.459471    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3ff065c-81fb-4272-b8e1-0b8213ecbd19-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.648892 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.648857    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr"]
Oct 30 23:03:36.662650 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.662624    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-80ecb3-5d98c75577-6g7jr"]
Oct 30 23:03:36.671561 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.671504    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:36.776006 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.775975    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3ff065c-81fb-4272-b8e1-0b8213ecbd19" path="/var/lib/kubelet/pods/a3ff065c-81fb-4272-b8e1-0b8213ecbd19/volumes"
Oct 30 23:03:36.776479 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.776463    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1cef895-22cd-4599-8239-d2ebf69cd234" path="/var/lib/kubelet/pods/d1cef895-22cd-4599-8239-d2ebf69cd234/volumes"
Oct 30 23:03:36.855643 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.855610    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:36.978519 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:36.978406    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l" podUID="7967e3f2-562e-476a-8e1e-294926499cee" containerName="main" containerID="cri-o://4ff87db5ab76260f525f3e4b56e1c424a067f4665ed892424bc0fc6faefc81da" gracePeriod=45
Oct 30 23:03:37.287431 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:37.287389    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:37.309010 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:37.308984    4889 generic.go:356] "Generic (PLEG): container finished" podID="7967e3f2-562e-476a-8e1e-294926499cee" containerID="4ff87db5ab76260f525f3e4b56e1c424a067f4665ed892424bc0fc6faefc81da" exitCode=0
Oct 30 23:03:37.309151 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:37.309057    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l" event={"ID":"7967e3f2-562e-476a-8e1e-294926499cee","Type":"ContainerDied","Data":"4ff87db5ab76260f525f3e4b56e1c424a067f4665ed892424bc0fc6faefc81da"}
Oct 30 23:03:37.309151 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:37.309100    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l" event={"ID":"7967e3f2-562e-476a-8e1e-294926499cee","Type":"ContainerDied","Data":"d462433fb4035d95e2f38eee6ce4dda779cfc10ff4df3ab105ff44600c797f12"}
Oct 30 23:03:37.309151 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:37.309113    4889 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d462433fb4035d95e2f38eee6ce4dda779cfc10ff4df3ab105ff44600c797f12"
Oct 30 23:03:37.319803 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:37.319775    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"
Oct 30 23:03:37.383333 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:37.383303    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-home\" (UniqueName: \"kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-home\") pod \"7967e3f2-562e-476a-8e1e-294926499cee\" (UID: \"7967e3f2-562e-476a-8e1e-294926499cee\") "
Oct 30 23:03:37.383333 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:37.383339    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-dshm\") pod \"7967e3f2-562e-476a-8e1e-294926499cee\" (UID: \"7967e3f2-562e-476a-8e1e-294926499cee\") "
Oct 30 23:03:37.383575 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:37.383387    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7967e3f2-562e-476a-8e1e-294926499cee-tls-certs\" (UniqueName: \"kubernetes.io/secret/7967e3f2-562e-476a-8e1e-294926499cee-tls-certs\") pod \"7967e3f2-562e-476a-8e1e-294926499cee\" (UID: \"7967e3f2-562e-476a-8e1e-294926499cee\") "
Oct 30 23:03:37.383575 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:37.383419    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-tmp-dir\") pod \"7967e3f2-562e-476a-8e1e-294926499cee\" (UID: \"7967e3f2-562e-476a-8e1e-294926499cee\") "
Oct 30 23:03:37.383575 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:37.383479    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-model-cache\") pod \"7967e3f2-562e-476a-8e1e-294926499cee\" (UID: \"7967e3f2-562e-476a-8e1e-294926499cee\") "
Oct 30 23:03:37.383575 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:37.383536    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7967e3f2-562e-476a-8e1e-294926499cee-kube-api-access-qhhg8\" (UniqueName: \"kubernetes.io/projected/7967e3f2-562e-476a-8e1e-294926499cee-kube-api-access-qhhg8\") pod \"7967e3f2-562e-476a-8e1e-294926499cee\" (UID: \"7967e3f2-562e-476a-8e1e-294926499cee\") "
Oct 30 23:03:37.383809 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:37.383625    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-home" pod "7967e3f2-562e-476a-8e1e-294926499cee" (UID: "7967e3f2-562e-476a-8e1e-294926499cee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:37.383809 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:37.383754    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-tmp-dir" pod "7967e3f2-562e-476a-8e1e-294926499cee" (UID: "7967e3f2-562e-476a-8e1e-294926499cee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:37.383809 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:37.383796    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-model-cache" pod "7967e3f2-562e-476a-8e1e-294926499cee" (UID: "7967e3f2-562e-476a-8e1e-294926499cee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:37.384032 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:37.384010    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:37.384032 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:37.384028    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:37.384032 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:37.384038    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:37.385943 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:37.385916    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-dshm" pod "7967e3f2-562e-476a-8e1e-294926499cee" (UID: "7967e3f2-562e-476a-8e1e-294926499cee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:37.386084 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:37.385947    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7967e3f2-562e-476a-8e1e-294926499cee-kube-api-access-qhhg8" pod "7967e3f2-562e-476a-8e1e-294926499cee" (UID: "7967e3f2-562e-476a-8e1e-294926499cee"). InnerVolumeSpecName "kube-api-access-qhhg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:37.386084 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:37.386004    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7967e3f2-562e-476a-8e1e-294926499cee-tls-certs" pod "7967e3f2-562e-476a-8e1e-294926499cee" (UID: "7967e3f2-562e-476a-8e1e-294926499cee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:37.485110 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:37.485078    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhhg8\" (UniqueName: \"kubernetes.io/projected/7967e3f2-562e-476a-8e1e-294926499cee-kube-api-access-qhhg8\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:37.485110 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:37.485105    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7967e3f2-562e-476a-8e1e-294926499cee-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:37.485110 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:37.485118    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7967e3f2-562e-476a-8e1e-294926499cee-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:37.689936 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:37.689813    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv" podUID="a5031ee6-029b-44a4-8c97-68a38a1d12d4" containerName="payload-processing" containerID="cri-o://d1868142ecb0b03833ecdd7ea9169196cbf8a5f116317a138bc986035cbabda9" gracePeriod=25
Oct 30 23:03:37.724435 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:37.724398    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv" podUID="d131b55f-ea6b-4f31-8053-fe81f0c0527e" containerName="payload-pre-processing" containerID="cri-o://1d8c057bbf32c5644bf5f293ee973b4ec7c0609ef55dfd177698287d52a94534" gracePeriod=25
Oct 30 23:03:38.297792 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:38.297754    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l" podUID="7967e3f2-562e-476a-8e1e-294926499cee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.243:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:03:38.317874 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:38.317843    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"
Oct 30 23:03:38.379335 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:38.379305    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"]
Oct 30 23:03:38.392935 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:38.392905    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-23aab2/sub-test-model-23aab2-kserve-d584986c5-f276l"]
Oct 30 23:03:38.751025 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:38.750996    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:38.776436 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:38.776404    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7967e3f2-562e-476a-8e1e-294926499cee" path="/var/lib/kubelet/pods/7967e3f2-562e-476a-8e1e-294926499cee/volumes"
Oct 30 23:03:39.275287 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:39.275247    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc"]
Oct 30 23:03:39.308531 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:39.308500    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k"]
Oct 30 23:03:39.428461 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:39.428417    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc" podUID="2e0a2179-dbbd-49c3-b6af-90aec117a44c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:03:39.460822 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:39.460788    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:39.645052 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:39.644965    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:40.105051 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.102256    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:40.265735 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.265697    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:40.307249 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:03:40.305368    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:03:40.393804 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.393725    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt"]
Oct 30 23:03:40.394353 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394338    4889 cpu_manager.go:414] "containerMap: removing container" podUID="7cfa769f-0310-4749-a929-646c9ab9bea5" containerName="istio-proxy"
Oct 30 23:03:40.394353 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394354    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cfa769f-0310-4749-a929-646c9ab9bea5" containerName="istio-proxy"
Oct 30 23:03:40.394468 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394364    4889 cpu_manager.go:414] "containerMap: removing container" podUID="fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4" containerName="istio-proxy"
Oct 30 23:03:40.394468 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394370    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4" containerName="istio-proxy"
Oct 30 23:03:40.394468 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394383    4889 cpu_manager.go:414] "containerMap: removing container" podUID="3d913d76-fad1-4092-bab7-e13c74cd2e0f" containerName="payload-pre-processing"
Oct 30 23:03:40.394468 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394388    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d913d76-fad1-4092-bab7-e13c74cd2e0f" containerName="payload-pre-processing"
Oct 30 23:03:40.394468 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394401    4889 cpu_manager.go:414] "containerMap: removing container" podUID="1aa5f50b-7768-46fd-9ce4-76c47268bce2" containerName="istio-proxy"
Oct 30 23:03:40.394468 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394408    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa5f50b-7768-46fd-9ce4-76c47268bce2" containerName="istio-proxy"
Oct 30 23:03:40.394468 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394423    4889 cpu_manager.go:414] "containerMap: removing container" podUID="94b31d20-a647-435f-8cdb-9d884c864207" containerName="payload-processing"
Oct 30 23:03:40.394468 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394430    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94b31d20-a647-435f-8cdb-9d884c864207" containerName="payload-processing"
Oct 30 23:03:40.394468 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394436    4889 cpu_manager.go:414] "containerMap: removing container" podUID="0589bfca-27f5-4ba4-93af-9a0ffd07ffb6" containerName="maas-api"
Oct 30 23:03:40.394468 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394440    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0589bfca-27f5-4ba4-93af-9a0ffd07ffb6" containerName="maas-api"
Oct 30 23:03:40.394468 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394450    4889 cpu_manager.go:414] "containerMap: removing container" podUID="63b2f029-b001-48d1-8c60-237096ebe16e" containerName="payload-pre-processing"
Oct 30 23:03:40.394468 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394454    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63b2f029-b001-48d1-8c60-237096ebe16e" containerName="payload-pre-processing"
Oct 30 23:03:40.394468 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394468    4889 cpu_manager.go:414] "containerMap: removing container" podUID="457f9b6e-69a8-49d4-b65a-5afe56213f23" containerName="maas-api"
Oct 30 23:03:40.394468 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394473    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="457f9b6e-69a8-49d4-b65a-5afe56213f23" containerName="maas-api"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394482    4889 cpu_manager.go:414] "containerMap: removing container" podUID="15e7f976-2726-4949-b6f7-31d4a362a5d4" containerName="test-curl-7269-09466f"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394486    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15e7f976-2726-4949-b6f7-31d4a362a5d4" containerName="test-curl-7269-09466f"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394500    4889 cpu_manager.go:414] "containerMap: removing container" podUID="9683180a-1f65-48bb-a62c-c78cef5799a3" containerName="payload-processing"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394505    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9683180a-1f65-48bb-a62c-c78cef5799a3" containerName="payload-processing"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394512    4889 cpu_manager.go:414] "containerMap: removing container" podUID="d1cef895-22cd-4599-8239-d2ebf69cd234" containerName="istio-proxy"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394516    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1cef895-22cd-4599-8239-d2ebf69cd234" containerName="istio-proxy"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394523    4889 cpu_manager.go:414] "containerMap: removing container" podUID="6881b170-1a5e-4c71-8fc9-c45349847afd" containerName="payload-pre-processing"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394528    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6881b170-1a5e-4c71-8fc9-c45349847afd" containerName="payload-pre-processing"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394536    4889 cpu_manager.go:414] "containerMap: removing container" podUID="8cf81220-6bf3-42ab-8330-ed9a73dc8190" containerName="payload-processing"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394541    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cf81220-6bf3-42ab-8330-ed9a73dc8190" containerName="payload-processing"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394546    4889 cpu_manager.go:414] "containerMap: removing container" podUID="2ace3d4c-11b6-41ca-a276-37746b3791fa" containerName="payload-processing"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394551    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ace3d4c-11b6-41ca-a276-37746b3791fa" containerName="payload-processing"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394560    4889 cpu_manager.go:414] "containerMap: removing container" podUID="c27dca2f-2be7-4a83-bdc0-9df7082f0027" containerName="payload-processing"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394564    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c27dca2f-2be7-4a83-bdc0-9df7082f0027" containerName="payload-processing"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394571    4889 cpu_manager.go:414] "containerMap: removing container" podUID="1511ead8-a673-47c2-9c21-9078d9e051ed" containerName="payload-pre-processing"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394575    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1511ead8-a673-47c2-9c21-9078d9e051ed" containerName="payload-pre-processing"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394582    4889 cpu_manager.go:414] "containerMap: removing container" podUID="d2279298-a1a4-469d-a188-f41478b6a598" containerName="istio-proxy"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394587    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2279298-a1a4-469d-a188-f41478b6a598" containerName="istio-proxy"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394594    4889 cpu_manager.go:414] "containerMap: removing container" podUID="e0a01454-5acd-417b-9e3c-62c826ef6558" containerName="payload-pre-processing"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394599    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0a01454-5acd-417b-9e3c-62c826ef6558" containerName="payload-pre-processing"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394606    4889 cpu_manager.go:414] "containerMap: removing container" podUID="ba4dc809-cd8c-4957-ade8-9d9d434e20c9" containerName="payload-processing"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394612    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba4dc809-cd8c-4957-ade8-9d9d434e20c9" containerName="payload-processing"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394618    4889 cpu_manager.go:414] "containerMap: removing container" podUID="d2279298-a1a4-469d-a188-f41478b6a598" containerName="istio-proxy"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394623    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2279298-a1a4-469d-a188-f41478b6a598" containerName="istio-proxy"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394630    4889 cpu_manager.go:414] "containerMap: removing container" podUID="7967e3f2-562e-476a-8e1e-294926499cee" containerName="main"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394634    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7967e3f2-562e-476a-8e1e-294926499cee" containerName="main"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394640    4889 cpu_manager.go:414] "containerMap: removing container" podUID="a3ff065c-81fb-4272-b8e1-0b8213ecbd19" containerName="maas-api"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394645    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ff065c-81fb-4272-b8e1-0b8213ecbd19" containerName="maas-api"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394653    4889 cpu_manager.go:414] "containerMap: removing container" podUID="1c1dcc08-1daf-4112-93d9-036d7dd70f6c" containerName="maas-api"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394657    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c1dcc08-1daf-4112-93d9-036d7dd70f6c" containerName="maas-api"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394669    4889 cpu_manager.go:414] "containerMap: removing container" podUID="1cd0f933-b208-424d-a9c1-04f8c6044054" containerName="main"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394674    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd0f933-b208-424d-a9c1-04f8c6044054" containerName="main"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394764    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="94b31d20-a647-435f-8cdb-9d884c864207" containerName="payload-processing"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394772    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="0589bfca-27f5-4ba4-93af-9a0ffd07ffb6" containerName="maas-api"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394778    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="63b2f029-b001-48d1-8c60-237096ebe16e" containerName="payload-pre-processing"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394784    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="457f9b6e-69a8-49d4-b65a-5afe56213f23" containerName="maas-api"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394792    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="15e7f976-2726-4949-b6f7-31d4a362a5d4" containerName="test-curl-7269-09466f"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394800    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="9683180a-1f65-48bb-a62c-c78cef5799a3" containerName="payload-processing"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394807    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cfa769f-0310-4749-a929-646c9ab9bea5" containerName="istio-proxy"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394812    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1cef895-22cd-4599-8239-d2ebf69cd234" containerName="istio-proxy"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394817    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="6881b170-1a5e-4c71-8fc9-c45349847afd" containerName="payload-pre-processing"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394824    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cf81220-6bf3-42ab-8330-ed9a73dc8190" containerName="payload-processing"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394829    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ace3d4c-11b6-41ca-a276-37746b3791fa" containerName="payload-processing"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394835    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="c27dca2f-2be7-4a83-bdc0-9df7082f0027" containerName="payload-processing"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394839    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="1511ead8-a673-47c2-9c21-9078d9e051ed" containerName="payload-pre-processing"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394844    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2279298-a1a4-469d-a188-f41478b6a598" containerName="istio-proxy"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394849    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0a01454-5acd-417b-9e3c-62c826ef6558" containerName="payload-pre-processing"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394854    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba4dc809-cd8c-4957-ade8-9d9d434e20c9" containerName="payload-processing"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394860    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2279298-a1a4-469d-a188-f41478b6a598" containerName="istio-proxy"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394865    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="7967e3f2-562e-476a-8e1e-294926499cee" containerName="main"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394870    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ff065c-81fb-4272-b8e1-0b8213ecbd19" containerName="maas-api"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394875    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa5f50b-7768-46fd-9ce4-76c47268bce2" containerName="istio-proxy"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394880    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c1dcc08-1daf-4112-93d9-036d7dd70f6c" containerName="maas-api"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394887    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd0f933-b208-424d-a9c1-04f8c6044054" containerName="main"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394893    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cfa769f-0310-4749-a929-646c9ab9bea5" containerName="istio-proxy"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394899    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb2ce23b-fd13-45c7-8e1d-f9be28fc05d4" containerName="istio-proxy"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394906    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d913d76-fad1-4092-bab7-e13c74cd2e0f" containerName="payload-pre-processing"
Oct 30 23:03:40.394895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.394912    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa5f50b-7768-46fd-9ce4-76c47268bce2" containerName="istio-proxy"
Oct 30 23:03:40.399530 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.399511    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt"
Oct 30 23:03:40.418098 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.418069    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt"]
Oct 30 23:03:40.426573 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.426522    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-bff950\""
Oct 30 23:03:40.426573 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.426535    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-bff950-dockercfg-cp8d5\""
Oct 30 23:03:40.436331 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.436297    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6" podUID="b812174d-906c-487b-91a9-7a1620aca44e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:03:40.536781 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.536737    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9nh7\" (UniqueName: \"kubernetes.io/projected/862bf26d-62d5-4a6f-ba47-242712b43d0a-kube-api-access-b9nh7\") pod \"payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt\" (UID: \"862bf26d-62d5-4a6f-ba47-242712b43d0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt"
Oct 30 23:03:40.536963 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.536929    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862bf26d-62d5-4a6f-ba47-242712b43d0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt\" (UID: \"862bf26d-62d5-4a6f-ba47-242712b43d0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt"
Oct 30 23:03:40.564057 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.564026    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt"]
Oct 30 23:03:40.564760 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.564742    4889 cpu_manager.go:414] "containerMap: removing container" podUID="7cfa769f-0310-4749-a929-646c9ab9bea5" containerName="istio-proxy"
Oct 30 23:03:40.564834 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.564762    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cfa769f-0310-4749-a929-646c9ab9bea5" containerName="istio-proxy"
Oct 30 23:03:40.564834 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.564779    4889 cpu_manager.go:414] "containerMap: removing container" podUID="1aa5f50b-7768-46fd-9ce4-76c47268bce2" containerName="istio-proxy"
Oct 30 23:03:40.564834 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.564784    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa5f50b-7768-46fd-9ce4-76c47268bce2" containerName="istio-proxy"
Oct 30 23:03:40.568450 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.568434    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt"
Oct 30 23:03:40.584444 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.584420    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt"]
Oct 30 23:03:40.638042 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.638009    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9nh7\" (UniqueName: \"kubernetes.io/projected/862bf26d-62d5-4a6f-ba47-242712b43d0a-kube-api-access-b9nh7\") pod \"payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt\" (UID: \"862bf26d-62d5-4a6f-ba47-242712b43d0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt"
Oct 30 23:03:40.638257 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.638082    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt\" (UID: \"bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt"
Oct 30 23:03:40.638257 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.638136    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm4md\" (UniqueName: \"kubernetes.io/projected/bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048-kube-api-access-rm4md\") pod \"payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt\" (UID: \"bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt"
Oct 30 23:03:40.638257 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.638157    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862bf26d-62d5-4a6f-ba47-242712b43d0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt\" (UID: \"862bf26d-62d5-4a6f-ba47-242712b43d0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt"
Oct 30 23:03:40.638886 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.638861    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862bf26d-62d5-4a6f-ba47-242712b43d0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt\" (UID: \"862bf26d-62d5-4a6f-ba47-242712b43d0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt"
Oct 30 23:03:40.659186 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.659125    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9nh7\" (UniqueName: \"kubernetes.io/projected/862bf26d-62d5-4a6f-ba47-242712b43d0a-kube-api-access-b9nh7\") pod \"payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt\" (UID: \"862bf26d-62d5-4a6f-ba47-242712b43d0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt"
Oct 30 23:03:40.715018 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.714993    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt"
Oct 30 23:03:40.740441 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.740414    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt\" (UID: \"bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt"
Oct 30 23:03:40.740570 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.740495    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm4md\" (UniqueName: \"kubernetes.io/projected/bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048-kube-api-access-rm4md\") pod \"payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt\" (UID: \"bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt"
Oct 30 23:03:40.742591 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.742249    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt\" (UID: \"bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt"
Oct 30 23:03:40.761776 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.761745    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm4md\" (UniqueName: \"kubernetes.io/projected/bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048-kube-api-access-rm4md\") pod \"payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt\" (UID: \"bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt"
Oct 30 23:03:40.884520 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:40.884485    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt"
Oct 30 23:03:41.039203 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:41.039171    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt"]
Oct 30 23:03:41.039882 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:03:41.039854    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod862bf26d_62d5_4a6f_ba47_242712b43d0a.slice/crio-bf46df764cca4a043cf48e900d4936d3b692ac3b16b86ebbef2b7363f47ba953 WatchSource:0}: Error finding container bf46df764cca4a043cf48e900d4936d3b692ac3b16b86ebbef2b7363f47ba953: Status 404 returned error can't find the container with id bf46df764cca4a043cf48e900d4936d3b692ac3b16b86ebbef2b7363f47ba953
Oct 30 23:03:41.191168 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:03:41.191102    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7bf3b1_7a2a_4bc6_82d4_9a28c193c048.slice/crio-ae308c2e16a0dc39e2997433ac138d27c820869282c5eafbb5b26a1c5329da68 WatchSource:0}: Error finding container ae308c2e16a0dc39e2997433ac138d27c820869282c5eafbb5b26a1c5329da68: Status 404 returned error can't find the container with id ae308c2e16a0dc39e2997433ac138d27c820869282c5eafbb5b26a1c5329da68
Oct 30 23:03:41.191574 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:41.191551    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt"]
Oct 30 23:03:41.237950 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:41.237917    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:41.373738 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:41.373695    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt" event={"ID":"bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048","Type":"ContainerStarted","Data":"75c6e610d15f0fe7228dcd0cc2dc8064165a72af7290f8e476e72229edf71209"}
Oct 30 23:03:41.374017 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:41.373790    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt" event={"ID":"bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048","Type":"ContainerStarted","Data":"ae308c2e16a0dc39e2997433ac138d27c820869282c5eafbb5b26a1c5329da68"}
Oct 30 23:03:41.374017 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:41.374000    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt"
Oct 30 23:03:41.375783 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:41.375754    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt" event={"ID":"862bf26d-62d5-4a6f-ba47-242712b43d0a","Type":"ContainerStarted","Data":"992dbf6f54a6c6ad1a9714ea906316fa62c46afbaa0fc2fdb67960c666e3d80a"}
Oct 30 23:03:41.375941 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:41.375801    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt" event={"ID":"862bf26d-62d5-4a6f-ba47-242712b43d0a","Type":"ContainerStarted","Data":"bf46df764cca4a043cf48e900d4936d3b692ac3b16b86ebbef2b7363f47ba953"}
Oct 30 23:03:41.375941 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:41.375858    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt"
Oct 30 23:03:41.417212 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:41.417159    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt" podStartSLOduration=1.417146684 podStartE2EDuration="1.417146684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:03:41.4159718 +0000 UTC m=+3406.772338115" watchObservedRunningTime="2026-10-30 23:03:41.417146684 +0000 UTC m=+3406.773512999"
Oct 30 23:03:41.462684 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:41.462586    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt" podStartSLOduration=1.462569015 podStartE2EDuration="1.462569015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:03:41.460554079 +0000 UTC m=+3406.816920394" watchObservedRunningTime="2026-10-30 23:03:41.462569015 +0000 UTC m=+3406.818935333"
Oct 30 23:03:41.916407 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:41.916373    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:42.077586 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:42.077543    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:42.991877 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:42.991842    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:43.435640 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:43.435551    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv" podUID="a5031ee6-029b-44a4-8c97-68a38a1d12d4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:03:43.435832 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:43.435655    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv" podUID="d131b55f-ea6b-4f31-8053-fe81f0c0527e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:03:43.553946 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:43.553901    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:43.734103 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:43.733977    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:43.833175 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:43.833141    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"]
Oct 30 23:03:43.833405 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:43.833359    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n" podUID="9833755d-9e30-41f6-806f-688a9cb72349" containerName="istio-proxy" containerID="cri-o://e9ddae6b8ec8a39d5acccee392f68000c1d6d6481c28cadb5f98ac92eb91b5c8" gracePeriod=30
Oct 30 23:03:44.054375 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:44.054345    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:44.275977 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:44.275934    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc" podUID="368f6471-b984-4935-92e6-047c8f81053f" containerName="payload-processing" containerID="cri-o://bb25d1a297429fc084984b461e011fcaa9b45b2995679e8d2e475ec02fe88956" gracePeriod=25
Oct 30 23:03:44.309577 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:44.309500    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k" podUID="2de71837-99a5-4e01-ae12-385e722759eb" containerName="payload-pre-processing" containerID="cri-o://f9de8cfaec16dc8ce77ddc7fd433efc4209f7a53eed48341dd0befa46505c4b5" gracePeriod=25
Oct 30 23:03:44.518093 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:44.518051    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:44.825301 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:44.825263    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx" podUID="cc0df870-2a4a-404b-9678-bdc054dce3e3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:03:45.110827 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:45.110719    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:45.279578 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:45.279544    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:45.724128 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:45.724092    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:46.756875 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:46.756842    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs"]
Oct 30 23:03:46.757342 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:46.757111    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs" podUID="ac794af8-6cd6-473f-bee9-3ea457729889" containerName="maas-api" containerID="cri-o://6204d8fc1cde0c397b2b2602aa4c0cdab2e2317c849be2d534b09904e3a56a2b" gracePeriod=30
Oct 30 23:03:46.876023 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:46.875989    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:47.071853 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:47.071824    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs"
Oct 30 23:03:47.226687 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:47.226650    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac794af8-6cd6-473f-bee9-3ea457729889-kube-api-access-hdnwd\" (UniqueName: \"kubernetes.io/projected/ac794af8-6cd6-473f-bee9-3ea457729889-kube-api-access-hdnwd\") pod \"ac794af8-6cd6-473f-bee9-3ea457729889\" (UID: \"ac794af8-6cd6-473f-bee9-3ea457729889\") "
Oct 30 23:03:47.226981 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:47.226768    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac794af8-6cd6-473f-bee9-3ea457729889-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac794af8-6cd6-473f-bee9-3ea457729889-maas-api-tls\") pod \"ac794af8-6cd6-473f-bee9-3ea457729889\" (UID: \"ac794af8-6cd6-473f-bee9-3ea457729889\") "
Oct 30 23:03:47.226981 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:47.226819    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac794af8-6cd6-473f-bee9-3ea457729889-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac794af8-6cd6-473f-bee9-3ea457729889-openshift-service-ca\") pod \"ac794af8-6cd6-473f-bee9-3ea457729889\" (UID: \"ac794af8-6cd6-473f-bee9-3ea457729889\") "
Oct 30 23:03:47.226981 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:47.226920    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac794af8-6cd6-473f-bee9-3ea457729889-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac794af8-6cd6-473f-bee9-3ea457729889-metrics-certs\") pod \"ac794af8-6cd6-473f-bee9-3ea457729889\" (UID: \"ac794af8-6cd6-473f-bee9-3ea457729889\") "
Oct 30 23:03:47.228481 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:47.228451    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac794af8-6cd6-473f-bee9-3ea457729889-openshift-service-ca" pod "ac794af8-6cd6-473f-bee9-3ea457729889" (UID: "ac794af8-6cd6-473f-bee9-3ea457729889"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:47.231270 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:47.230892    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac794af8-6cd6-473f-bee9-3ea457729889-metrics-certs" pod "ac794af8-6cd6-473f-bee9-3ea457729889" (UID: "ac794af8-6cd6-473f-bee9-3ea457729889"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:47.231270 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:47.230954    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac794af8-6cd6-473f-bee9-3ea457729889-maas-api-tls" pod "ac794af8-6cd6-473f-bee9-3ea457729889" (UID: "ac794af8-6cd6-473f-bee9-3ea457729889"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:47.231270 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:47.231025    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac794af8-6cd6-473f-bee9-3ea457729889-kube-api-access-hdnwd" pod "ac794af8-6cd6-473f-bee9-3ea457729889" (UID: "ac794af8-6cd6-473f-bee9-3ea457729889"). InnerVolumeSpecName "kube-api-access-hdnwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:47.329297 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:47.329196    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac794af8-6cd6-473f-bee9-3ea457729889-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:47.329297 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:47.329242    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdnwd\" (UniqueName: \"kubernetes.io/projected/ac794af8-6cd6-473f-bee9-3ea457729889-kube-api-access-hdnwd\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:47.329297 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:47.329252    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac794af8-6cd6-473f-bee9-3ea457729889-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:47.329297 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:47.329261    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac794af8-6cd6-473f-bee9-3ea457729889-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:47.418898 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:47.418868    4889 generic.go:356] "Generic (PLEG): container finished" podID="ac794af8-6cd6-473f-bee9-3ea457729889" containerID="6204d8fc1cde0c397b2b2602aa4c0cdab2e2317c849be2d534b09904e3a56a2b" exitCode=0
Oct 30 23:03:47.419089 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:47.418915    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs"
Oct 30 23:03:47.419089 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:47.418955    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs" event={"ID":"ac794af8-6cd6-473f-bee9-3ea457729889","Type":"ContainerDied","Data":"6204d8fc1cde0c397b2b2602aa4c0cdab2e2317c849be2d534b09904e3a56a2b"}
Oct 30 23:03:47.419089 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:47.419010    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs" event={"ID":"ac794af8-6cd6-473f-bee9-3ea457729889","Type":"ContainerDied","Data":"19fef0741fa185e1f7c045a164eae2db656c3b0d128be9543f1f380adb7e56e2"}
Oct 30 23:03:47.419089 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:47.419035    4889 scope.go:122] "RemoveContainer" containerID="6204d8fc1cde0c397b2b2602aa4c0cdab2e2317c849be2d534b09904e3a56a2b"
Oct 30 23:03:47.431058 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:47.431035    4889 scope.go:122] "RemoveContainer" containerID="6204d8fc1cde0c397b2b2602aa4c0cdab2e2317c849be2d534b09904e3a56a2b"
Oct 30 23:03:47.431373 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:03:47.431352    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6204d8fc1cde0c397b2b2602aa4c0cdab2e2317c849be2d534b09904e3a56a2b\": container with ID starting with 6204d8fc1cde0c397b2b2602aa4c0cdab2e2317c849be2d534b09904e3a56a2b not found: ID does not exist" containerID="6204d8fc1cde0c397b2b2602aa4c0cdab2e2317c849be2d534b09904e3a56a2b"
Oct 30 23:03:47.431443 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:47.431382    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6204d8fc1cde0c397b2b2602aa4c0cdab2e2317c849be2d534b09904e3a56a2b"} err="failed to get container status \"6204d8fc1cde0c397b2b2602aa4c0cdab2e2317c849be2d534b09904e3a56a2b\": rpc error: code = NotFound desc = could not find container \"6204d8fc1cde0c397b2b2602aa4c0cdab2e2317c849be2d534b09904e3a56a2b\": container with ID starting with 6204d8fc1cde0c397b2b2602aa4c0cdab2e2317c849be2d534b09904e3a56a2b not found: ID does not exist"
Oct 30 23:03:47.476801 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:47.476771    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs"]
Oct 30 23:03:47.489941 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:47.489913    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bed544-6585dd9b99-k49xs"]
Oct 30 23:03:47.525855 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:47.525819    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:47.709115 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:47.709020    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:47.875146 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:47.875103    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc" podUID="368f6471-b984-4935-92e6-047c8f81053f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:03:47.875612 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:47.875149    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k" podUID="2de71837-99a5-4e01-ae12-385e722759eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:03:48.187904 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:48.187869    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:48.340707 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:48.340678    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx"
Oct 30 23:03:48.428489 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:48.428455    4889 generic.go:356] "Generic (PLEG): container finished" podID="cc0df870-2a4a-404b-9678-bdc054dce3e3" containerID="dc70155ffafe9e975a8ca507d202284df36f0ae94460753a5267febbe8c22293" exitCode=0
Oct 30 23:03:48.428670 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:48.428528    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx"
Oct 30 23:03:48.428670 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:48.428541    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx" event={"ID":"cc0df870-2a4a-404b-9678-bdc054dce3e3","Type":"ContainerDied","Data":"dc70155ffafe9e975a8ca507d202284df36f0ae94460753a5267febbe8c22293"}
Oct 30 23:03:48.428670 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:48.428607    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx" event={"ID":"cc0df870-2a4a-404b-9678-bdc054dce3e3","Type":"ContainerDied","Data":"a0716887e39b93d26bfbbca2c5dd96fdf78023de42306b53213e77d3f98e983a"}
Oct 30 23:03:48.428670 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:48.428646    4889 scope.go:122] "RemoveContainer" containerID="dc70155ffafe9e975a8ca507d202284df36f0ae94460753a5267febbe8c22293"
Oct 30 23:03:48.439187 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:48.439170    4889 scope.go:122] "RemoveContainer" containerID="dc70155ffafe9e975a8ca507d202284df36f0ae94460753a5267febbe8c22293"
Oct 30 23:03:48.439528 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:03:48.439509    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc70155ffafe9e975a8ca507d202284df36f0ae94460753a5267febbe8c22293\": container with ID starting with dc70155ffafe9e975a8ca507d202284df36f0ae94460753a5267febbe8c22293 not found: ID does not exist" containerID="dc70155ffafe9e975a8ca507d202284df36f0ae94460753a5267febbe8c22293"
Oct 30 23:03:48.439582 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:48.439540    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc70155ffafe9e975a8ca507d202284df36f0ae94460753a5267febbe8c22293"} err="failed to get container status \"dc70155ffafe9e975a8ca507d202284df36f0ae94460753a5267febbe8c22293\": rpc error: code = NotFound desc = could not find container \"dc70155ffafe9e975a8ca507d202284df36f0ae94460753a5267febbe8c22293\": container with ID starting with dc70155ffafe9e975a8ca507d202284df36f0ae94460753a5267febbe8c22293 not found: ID does not exist"
Oct 30 23:03:48.444098 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:48.444078    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc0df870-2a4a-404b-9678-bdc054dce3e3-kube-api-access-6lt56\" (UniqueName: \"kubernetes.io/projected/cc0df870-2a4a-404b-9678-bdc054dce3e3-kube-api-access-6lt56\") pod \"cc0df870-2a4a-404b-9678-bdc054dce3e3\" (UID: \"cc0df870-2a4a-404b-9678-bdc054dce3e3\") "
Oct 30 23:03:48.444181 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:48.444149    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc0df870-2a4a-404b-9678-bdc054dce3e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0df870-2a4a-404b-9678-bdc054dce3e3-plugins-config-volume\") pod \"cc0df870-2a4a-404b-9678-bdc054dce3e3\" (UID: \"cc0df870-2a4a-404b-9678-bdc054dce3e3\") "
Oct 30 23:03:48.444457 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:48.444436    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc0df870-2a4a-404b-9678-bdc054dce3e3-plugins-config-volume" pod "cc0df870-2a4a-404b-9678-bdc054dce3e3" (UID: "cc0df870-2a4a-404b-9678-bdc054dce3e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:48.444543 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:48.444529    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0df870-2a4a-404b-9678-bdc054dce3e3-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:48.446676 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:48.446654    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0df870-2a4a-404b-9678-bdc054dce3e3-kube-api-access-6lt56" pod "cc0df870-2a4a-404b-9678-bdc054dce3e3" (UID: "cc0df870-2a4a-404b-9678-bdc054dce3e3"). InnerVolumeSpecName "kube-api-access-6lt56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:48.545325 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:48.545285    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lt56\" (UniqueName: \"kubernetes.io/projected/cc0df870-2a4a-404b-9678-bdc054dce3e3-kube-api-access-6lt56\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:48.775469 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:48.775437    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac794af8-6cd6-473f-bee9-3ea457729889" path="/var/lib/kubelet/pods/ac794af8-6cd6-473f-bee9-3ea457729889/volumes"
Oct 30 23:03:48.777416 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:48.777384    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx"]
Oct 30 23:03:48.790787 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:48.790761    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-80ecb3-7484c79f79-cndqx"]
Oct 30 23:03:49.119739 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.119711    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 23:03:49.252659 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.252610    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-workload-certs\") pod \"9833755d-9e30-41f6-806f-688a9cb72349\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") "
Oct 30 23:03:49.252865 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.252700    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9833755d-9e30-41f6-806f-688a9cb72349-istio-token\" (UniqueName: \"kubernetes.io/projected/9833755d-9e30-41f6-806f-688a9cb72349-istio-token\") pod \"9833755d-9e30-41f6-806f-688a9cb72349\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") "
Oct 30 23:03:49.252865 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.252745    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9833755d-9e30-41f6-806f-688a9cb72349-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9833755d-9e30-41f6-806f-688a9cb72349-istio-podinfo\") pod \"9833755d-9e30-41f6-806f-688a9cb72349\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") "
Oct 30 23:03:49.252865 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.252797    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-istio-data\") pod \"9833755d-9e30-41f6-806f-688a9cb72349\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") "
Oct 30 23:03:49.253038 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.252905    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9833755d-9e30-41f6-806f-688a9cb72349-kube-api-access-4cclv\" (UniqueName: \"kubernetes.io/projected/9833755d-9e30-41f6-806f-688a9cb72349-kube-api-access-4cclv\") pod \"9833755d-9e30-41f6-806f-688a9cb72349\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") "
Oct 30 23:03:49.253038 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.252930    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9833755d-9e30-41f6-806f-688a9cb72349-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9833755d-9e30-41f6-806f-688a9cb72349-istiod-ca-cert\") pod \"9833755d-9e30-41f6-806f-688a9cb72349\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") "
Oct 30 23:03:49.253038 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.252963    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-credential-socket\") pod \"9833755d-9e30-41f6-806f-688a9cb72349\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") "
Oct 30 23:03:49.253038 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.252962    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-workload-certs" pod "9833755d-9e30-41f6-806f-688a9cb72349" (UID: "9833755d-9e30-41f6-806f-688a9cb72349"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:49.253038 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.252986    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-istio-envoy\") pod \"9833755d-9e30-41f6-806f-688a9cb72349\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") "
Oct 30 23:03:49.253038 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.253024    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-workload-socket\") pod \"9833755d-9e30-41f6-806f-688a9cb72349\" (UID: \"9833755d-9e30-41f6-806f-688a9cb72349\") "
Oct 30 23:03:49.253433 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.253130    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-istio-data" pod "9833755d-9e30-41f6-806f-688a9cb72349" (UID: "9833755d-9e30-41f6-806f-688a9cb72349"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:49.253433 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.253320    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9833755d-9e30-41f6-806f-688a9cb72349-istiod-ca-cert" pod "9833755d-9e30-41f6-806f-688a9cb72349" (UID: "9833755d-9e30-41f6-806f-688a9cb72349"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:49.253433 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.253410    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-credential-socket" pod "9833755d-9e30-41f6-806f-688a9cb72349" (UID: "9833755d-9e30-41f6-806f-688a9cb72349"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:49.253786 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.253584    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-workload-socket" pod "9833755d-9e30-41f6-806f-688a9cb72349" (UID: "9833755d-9e30-41f6-806f-688a9cb72349"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:49.253786 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.253616    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9833755d-9e30-41f6-806f-688a9cb72349-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:49.253786 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.253640    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:49.253786 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.253656    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:49.253786 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.253670    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:49.255280 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.255254    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9833755d-9e30-41f6-806f-688a9cb72349-istio-podinfo" pod "9833755d-9e30-41f6-806f-688a9cb72349" (UID: "9833755d-9e30-41f6-806f-688a9cb72349"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:03:49.255425 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.255331    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9833755d-9e30-41f6-806f-688a9cb72349-istio-token" pod "9833755d-9e30-41f6-806f-688a9cb72349" (UID: "9833755d-9e30-41f6-806f-688a9cb72349"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:49.255425 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.255374    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-istio-envoy" pod "9833755d-9e30-41f6-806f-688a9cb72349" (UID: "9833755d-9e30-41f6-806f-688a9cb72349"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:49.255619 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.255600    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9833755d-9e30-41f6-806f-688a9cb72349-kube-api-access-4cclv" pod "9833755d-9e30-41f6-806f-688a9cb72349" (UID: "9833755d-9e30-41f6-806f-688a9cb72349"). InnerVolumeSpecName "kube-api-access-4cclv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:49.354929 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.354886    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cclv\" (UniqueName: \"kubernetes.io/projected/9833755d-9e30-41f6-806f-688a9cb72349-kube-api-access-4cclv\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:49.354929 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.354921    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:49.355144 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.354935    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9833755d-9e30-41f6-806f-688a9cb72349-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:49.355144 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.354962    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9833755d-9e30-41f6-806f-688a9cb72349-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:49.355144 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.354974    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9833755d-9e30-41f6-806f-688a9cb72349-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:49.427972 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.427927    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc" podUID="2e0a2179-dbbd-49c3-b6af-90aec117a44c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:03:49.438707 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.438676    4889 generic.go:356] "Generic (PLEG): container finished" podID="9833755d-9e30-41f6-806f-688a9cb72349" containerID="e9ddae6b8ec8a39d5acccee392f68000c1d6d6481c28cadb5f98ac92eb91b5c8" exitCode=0
Oct 30 23:03:49.438875 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.438732    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"
Oct 30 23:03:49.438875 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.438752    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n" event={"ID":"9833755d-9e30-41f6-806f-688a9cb72349","Type":"ContainerDied","Data":"e9ddae6b8ec8a39d5acccee392f68000c1d6d6481c28cadb5f98ac92eb91b5c8"}
Oct 30 23:03:49.438875 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.438810    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n" event={"ID":"9833755d-9e30-41f6-806f-688a9cb72349","Type":"ContainerDied","Data":"b2dcd0628fcb41be4bcb6ecba25d1ea488c8e089116dae001e409bc4c2587846"}
Oct 30 23:03:49.438875 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.438841    4889 scope.go:122] "RemoveContainer" containerID="e9ddae6b8ec8a39d5acccee392f68000c1d6d6481c28cadb5f98ac92eb91b5c8"
Oct 30 23:03:49.450035 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.450014    4889 scope.go:122] "RemoveContainer" containerID="da0e6ca092e95570b9a393d80f2b5144ef5056dd31dbe09c02b06910de65c043"
Oct 30 23:03:49.460159 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.460136    4889 scope.go:122] "RemoveContainer" containerID="e9ddae6b8ec8a39d5acccee392f68000c1d6d6481c28cadb5f98ac92eb91b5c8"
Oct 30 23:03:49.460490 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:03:49.460461    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9ddae6b8ec8a39d5acccee392f68000c1d6d6481c28cadb5f98ac92eb91b5c8\": container with ID starting with e9ddae6b8ec8a39d5acccee392f68000c1d6d6481c28cadb5f98ac92eb91b5c8 not found: ID does not exist" containerID="e9ddae6b8ec8a39d5acccee392f68000c1d6d6481c28cadb5f98ac92eb91b5c8"
Oct 30 23:03:49.460603 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.460500    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ddae6b8ec8a39d5acccee392f68000c1d6d6481c28cadb5f98ac92eb91b5c8"} err="failed to get container status \"e9ddae6b8ec8a39d5acccee392f68000c1d6d6481c28cadb5f98ac92eb91b5c8\": rpc error: code = NotFound desc = could not find container \"e9ddae6b8ec8a39d5acccee392f68000c1d6d6481c28cadb5f98ac92eb91b5c8\": container with ID starting with e9ddae6b8ec8a39d5acccee392f68000c1d6d6481c28cadb5f98ac92eb91b5c8 not found: ID does not exist"
Oct 30 23:03:49.460603 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.460525    4889 scope.go:122] "RemoveContainer" containerID="da0e6ca092e95570b9a393d80f2b5144ef5056dd31dbe09c02b06910de65c043"
Oct 30 23:03:49.460886 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:03:49.460826    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da0e6ca092e95570b9a393d80f2b5144ef5056dd31dbe09c02b06910de65c043\": container with ID starting with da0e6ca092e95570b9a393d80f2b5144ef5056dd31dbe09c02b06910de65c043 not found: ID does not exist" containerID="da0e6ca092e95570b9a393d80f2b5144ef5056dd31dbe09c02b06910de65c043"
Oct 30 23:03:49.460886 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.460862    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da0e6ca092e95570b9a393d80f2b5144ef5056dd31dbe09c02b06910de65c043"} err="failed to get container status \"da0e6ca092e95570b9a393d80f2b5144ef5056dd31dbe09c02b06910de65c043\": rpc error: code = NotFound desc = could not find container \"da0e6ca092e95570b9a393d80f2b5144ef5056dd31dbe09c02b06910de65c043\": container with ID starting with da0e6ca092e95570b9a393d80f2b5144ef5056dd31dbe09c02b06910de65c043 not found: ID does not exist"
Oct 30 23:03:49.495093 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.495061    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"]
Oct 30 23:03:49.508003 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.507982    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-2xr8n"]
Oct 30 23:03:49.659577 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:49.659482    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:50.435496 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:50.435459    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6" podUID="b812174d-906c-487b-91a9-7a1620aca44e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:03:50.775285 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:50.775249    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9833755d-9e30-41f6-806f-688a9cb72349" path="/var/lib/kubelet/pods/9833755d-9e30-41f6-806f-688a9cb72349/volumes"
Oct 30 23:03:50.775692 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:50.775678    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc0df870-2a4a-404b-9678-bdc054dce3e3" path="/var/lib/kubelet/pods/cc0df870-2a4a-404b-9678-bdc054dce3e3/volumes"
Oct 30 23:03:51.040845 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.040756    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main" containerID="cri-o://eccf8de25ed87a3c14921ff096bd70fd5580728bb2ee9c3e82014412963c69b1" gracePeriod=45
Oct 30 23:03:51.310211 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.310157    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 23:03:51.456616 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.456583    4889 generic.go:356] "Generic (PLEG): container finished" podID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerID="eccf8de25ed87a3c14921ff096bd70fd5580728bb2ee9c3e82014412963c69b1" exitCode=0
Oct 30 23:03:51.457094 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.456618    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" event={"ID":"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186","Type":"ContainerDied","Data":"eccf8de25ed87a3c14921ff096bd70fd5580728bb2ee9c3e82014412963c69b1"}
Oct 30 23:03:51.457094 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.456688    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd" event={"ID":"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186","Type":"ContainerDied","Data":"eb152f66dfb86ecc9845d82d560b3c2b1544ce9ccc95a3a268713b2df5f008a1"}
Oct 30 23:03:51.457094 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.456708    4889 scope.go:122] "RemoveContainer" containerID="eccf8de25ed87a3c14921ff096bd70fd5580728bb2ee9c3e82014412963c69b1"
Oct 30 23:03:51.457094 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.456642    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"
Oct 30 23:03:51.479891 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.479866    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-kube-api-access-8x6km\" (UniqueName: \"kubernetes.io/projected/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-kube-api-access-8x6km\") pod \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\" (UID: \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\") "
Oct 30 23:03:51.480003 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.479913    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-tls-certs\" (UniqueName: \"kubernetes.io/secret/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-tls-certs\") pod \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\" (UID: \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\") "
Oct 30 23:03:51.480003 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.479946    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-tmp-dir\") pod \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\" (UID: \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\") "
Oct 30 23:03:51.480003 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.479988    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-home\" (UniqueName: \"kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-home\") pod \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\" (UID: \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\") "
Oct 30 23:03:51.480174 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.480034    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-dshm\") pod \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\" (UID: \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\") "
Oct 30 23:03:51.480174 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.480110    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-model-cache\") pod \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\" (UID: \"dd0ddc98-49ec-4ca9-ba5d-1db4b785b186\") "
Oct 30 23:03:51.480360 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.480337    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-tmp-dir" pod "dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" (UID: "dd0ddc98-49ec-4ca9-ba5d-1db4b785b186"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:51.480435 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.480346    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-home" pod "dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" (UID: "dd0ddc98-49ec-4ca9-ba5d-1db4b785b186"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:51.480435 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.480381    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-model-cache" pod "dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" (UID: "dd0ddc98-49ec-4ca9-ba5d-1db4b785b186"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:51.480639 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.480614    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:51.480896 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.480642    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:51.480896 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.480650    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:51.482458 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.482431    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-kube-api-access-8x6km" pod "dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" (UID: "dd0ddc98-49ec-4ca9-ba5d-1db4b785b186"). InnerVolumeSpecName "kube-api-access-8x6km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:51.482546 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.482518    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-dshm" pod "dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" (UID: "dd0ddc98-49ec-4ca9-ba5d-1db4b785b186"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:51.482546 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.482526    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-tls-certs" pod "dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" (UID: "dd0ddc98-49ec-4ca9-ba5d-1db4b785b186"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:51.550594 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.550529    4889 scope.go:122] "RemoveContainer" containerID="eccf8de25ed87a3c14921ff096bd70fd5580728bb2ee9c3e82014412963c69b1"
Oct 30 23:03:51.551049 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:03:51.551011    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eccf8de25ed87a3c14921ff096bd70fd5580728bb2ee9c3e82014412963c69b1\": container with ID starting with eccf8de25ed87a3c14921ff096bd70fd5580728bb2ee9c3e82014412963c69b1 not found: ID does not exist" containerID="eccf8de25ed87a3c14921ff096bd70fd5580728bb2ee9c3e82014412963c69b1"
Oct 30 23:03:51.551182 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.551061    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eccf8de25ed87a3c14921ff096bd70fd5580728bb2ee9c3e82014412963c69b1"} err="failed to get container status \"eccf8de25ed87a3c14921ff096bd70fd5580728bb2ee9c3e82014412963c69b1\": rpc error: code = NotFound desc = could not find container \"eccf8de25ed87a3c14921ff096bd70fd5580728bb2ee9c3e82014412963c69b1\": container with ID starting with eccf8de25ed87a3c14921ff096bd70fd5580728bb2ee9c3e82014412963c69b1 not found: ID does not exist"
Oct 30 23:03:51.581921 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.581826    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x6km\" (UniqueName: \"kubernetes.io/projected/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-kube-api-access-8x6km\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:51.581921 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.581862    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:51.581921 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.581874    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:51.720625 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.720546    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:51.752273 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.752212    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6"
Oct 30 23:03:51.763133 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.763113    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc"
Oct 30 23:03:51.834284 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.834184    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"]
Oct 30 23:03:51.838203 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.838180    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-54114e/distinct-w5-54114e-kserve-6995667bf6-hjvzd"]
Oct 30 23:03:51.885603 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.885564    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e0a2179-dbbd-49c3-b6af-90aec117a44c-kube-api-access-qfcv8\" (UniqueName: \"kubernetes.io/projected/2e0a2179-dbbd-49c3-b6af-90aec117a44c-kube-api-access-qfcv8\") pod \"2e0a2179-dbbd-49c3-b6af-90aec117a44c\" (UID: \"2e0a2179-dbbd-49c3-b6af-90aec117a44c\") "
Oct 30 23:03:51.885806 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.885707    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e0a2179-dbbd-49c3-b6af-90aec117a44c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e0a2179-dbbd-49c3-b6af-90aec117a44c-plugins-config-volume\") pod \"2e0a2179-dbbd-49c3-b6af-90aec117a44c\" (UID: \"2e0a2179-dbbd-49c3-b6af-90aec117a44c\") "
Oct 30 23:03:51.885806 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.885787    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b812174d-906c-487b-91a9-7a1620aca44e-kube-api-access-zzq9p\" (UniqueName: \"kubernetes.io/projected/b812174d-906c-487b-91a9-7a1620aca44e-kube-api-access-zzq9p\") pod \"b812174d-906c-487b-91a9-7a1620aca44e\" (UID: \"b812174d-906c-487b-91a9-7a1620aca44e\") "
Oct 30 23:03:51.885940 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.885884    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b812174d-906c-487b-91a9-7a1620aca44e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b812174d-906c-487b-91a9-7a1620aca44e-plugins-config-volume\") pod \"b812174d-906c-487b-91a9-7a1620aca44e\" (UID: \"b812174d-906c-487b-91a9-7a1620aca44e\") "
Oct 30 23:03:51.887916 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.887526    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b812174d-906c-487b-91a9-7a1620aca44e-plugins-config-volume" pod "b812174d-906c-487b-91a9-7a1620aca44e" (UID: "b812174d-906c-487b-91a9-7a1620aca44e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:51.887916 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.887692    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e0a2179-dbbd-49c3-b6af-90aec117a44c-plugins-config-volume" pod "2e0a2179-dbbd-49c3-b6af-90aec117a44c" (UID: "2e0a2179-dbbd-49c3-b6af-90aec117a44c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:51.889602 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.889572    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e0a2179-dbbd-49c3-b6af-90aec117a44c-kube-api-access-qfcv8" pod "2e0a2179-dbbd-49c3-b6af-90aec117a44c" (UID: "2e0a2179-dbbd-49c3-b6af-90aec117a44c"). InnerVolumeSpecName "kube-api-access-qfcv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:51.890476 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.890454    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b812174d-906c-487b-91a9-7a1620aca44e-kube-api-access-zzq9p" pod "b812174d-906c-487b-91a9-7a1620aca44e" (UID: "b812174d-906c-487b-91a9-7a1620aca44e"). InnerVolumeSpecName "kube-api-access-zzq9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:51.987397 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.987365    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzq9p\" (UniqueName: \"kubernetes.io/projected/b812174d-906c-487b-91a9-7a1620aca44e-kube-api-access-zzq9p\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:51.987397 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.987393    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b812174d-906c-487b-91a9-7a1620aca44e-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:51.987397 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.987404    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfcv8\" (UniqueName: \"kubernetes.io/projected/2e0a2179-dbbd-49c3-b6af-90aec117a44c-kube-api-access-qfcv8\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:51.987397 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:51.987413    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e0a2179-dbbd-49c3-b6af-90aec117a44c-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:52.097708 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.097614    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:52.385736 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.385636    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt"
Oct 30 23:03:52.386071 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.386052    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt"
Oct 30 23:03:52.465478 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.465441    4889 generic.go:356] "Generic (PLEG): container finished" podID="2e0a2179-dbbd-49c3-b6af-90aec117a44c" containerID="dcf269360f829facc20b88c5d510c5292f50e8a0690a69f665911fb548b1f89e" exitCode=0
Oct 30 23:03:52.465999 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.465500    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc"
Oct 30 23:03:52.465999 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.465533    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc" event={"ID":"2e0a2179-dbbd-49c3-b6af-90aec117a44c","Type":"ContainerDied","Data":"dcf269360f829facc20b88c5d510c5292f50e8a0690a69f665911fb548b1f89e"}
Oct 30 23:03:52.465999 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.465592    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc" event={"ID":"2e0a2179-dbbd-49c3-b6af-90aec117a44c","Type":"ContainerDied","Data":"a31da173b660dfee9e31616f8bf439b90351997fcc017288b913729605519cab"}
Oct 30 23:03:52.465999 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.465624    4889 scope.go:122] "RemoveContainer" containerID="dcf269360f829facc20b88c5d510c5292f50e8a0690a69f665911fb548b1f89e"
Oct 30 23:03:52.467393 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.467366    4889 generic.go:356] "Generic (PLEG): container finished" podID="b812174d-906c-487b-91a9-7a1620aca44e" containerID="632093beb73e141eed51a24f65cd85fb6454fb25f32d92a7100dee5e129bbbb0" exitCode=0
Oct 30 23:03:52.467533 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.467504    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6" event={"ID":"b812174d-906c-487b-91a9-7a1620aca44e","Type":"ContainerDied","Data":"632093beb73e141eed51a24f65cd85fb6454fb25f32d92a7100dee5e129bbbb0"}
Oct 30 23:03:52.467608 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.467551    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6" event={"ID":"b812174d-906c-487b-91a9-7a1620aca44e","Type":"ContainerDied","Data":"b5afa9711a2ccc2f84d35e8a782a83d3d5cd94dbead18a010725f949089f0ff8"}
Oct 30 23:03:52.467671 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.467617    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6"
Oct 30 23:03:52.480123 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.480095    4889 scope.go:122] "RemoveContainer" containerID="dcf269360f829facc20b88c5d510c5292f50e8a0690a69f665911fb548b1f89e"
Oct 30 23:03:52.483466 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:03:52.483300    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcf269360f829facc20b88c5d510c5292f50e8a0690a69f665911fb548b1f89e\": container with ID starting with dcf269360f829facc20b88c5d510c5292f50e8a0690a69f665911fb548b1f89e not found: ID does not exist" containerID="dcf269360f829facc20b88c5d510c5292f50e8a0690a69f665911fb548b1f89e"
Oct 30 23:03:52.483466 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.483342    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcf269360f829facc20b88c5d510c5292f50e8a0690a69f665911fb548b1f89e"} err="failed to get container status \"dcf269360f829facc20b88c5d510c5292f50e8a0690a69f665911fb548b1f89e\": rpc error: code = NotFound desc = could not find container \"dcf269360f829facc20b88c5d510c5292f50e8a0690a69f665911fb548b1f89e\": container with ID starting with dcf269360f829facc20b88c5d510c5292f50e8a0690a69f665911fb548b1f89e not found: ID does not exist"
Oct 30 23:03:52.483466 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.483368    4889 scope.go:122] "RemoveContainer" containerID="632093beb73e141eed51a24f65cd85fb6454fb25f32d92a7100dee5e129bbbb0"
Oct 30 23:03:52.491175 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.490015    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:52.502408 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.502384    4889 scope.go:122] "RemoveContainer" containerID="632093beb73e141eed51a24f65cd85fb6454fb25f32d92a7100dee5e129bbbb0"
Oct 30 23:03:52.504491 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:03:52.504459    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"632093beb73e141eed51a24f65cd85fb6454fb25f32d92a7100dee5e129bbbb0\": container with ID starting with 632093beb73e141eed51a24f65cd85fb6454fb25f32d92a7100dee5e129bbbb0 not found: ID does not exist" containerID="632093beb73e141eed51a24f65cd85fb6454fb25f32d92a7100dee5e129bbbb0"
Oct 30 23:03:52.504715 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.504679    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"632093beb73e141eed51a24f65cd85fb6454fb25f32d92a7100dee5e129bbbb0"} err="failed to get container status \"632093beb73e141eed51a24f65cd85fb6454fb25f32d92a7100dee5e129bbbb0\": rpc error: code = NotFound desc = could not find container \"632093beb73e141eed51a24f65cd85fb6454fb25f32d92a7100dee5e129bbbb0\": container with ID starting with 632093beb73e141eed51a24f65cd85fb6454fb25f32d92a7100dee5e129bbbb0 not found: ID does not exist"
Oct 30 23:03:52.546451 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.546421    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6"]
Oct 30 23:03:52.552087 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.552056    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-012ba0-5669b88f55-swpg6"]
Oct 30 23:03:52.594019 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.593986    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc"]
Oct 30 23:03:52.596977 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.596951    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-012ba0-669ff7bfff-8w9bc"]
Oct 30 23:03:52.651653 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.651568    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:52.739758 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.739717    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"]
Oct 30 23:03:52.739963 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.739941    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg" podUID="c02298a7-464d-4cc4-8bef-3be1dfc6f3ab" containerName="istio-proxy" containerID="cri-o://be3cd57e007b64c0705a95159a6611725bdd672c86bd63fb3d1e9bc05d1cfc3b" gracePeriod=30
Oct 30 23:03:52.774719 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.774684    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e0a2179-dbbd-49c3-b6af-90aec117a44c" path="/var/lib/kubelet/pods/2e0a2179-dbbd-49c3-b6af-90aec117a44c/volumes"
Oct 30 23:03:52.775195 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.775173    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b812174d-906c-487b-91a9-7a1620aca44e" path="/var/lib/kubelet/pods/b812174d-906c-487b-91a9-7a1620aca44e/volumes"
Oct 30 23:03:52.775626 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.775612    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" path="/var/lib/kubelet/pods/dd0ddc98-49ec-4ca9-ba5d-1db4b785b186/volumes"
Oct 30 23:03:52.825743 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:52.825708    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:53.435410 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:53.435373    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv" podUID="d131b55f-ea6b-4f31-8053-fe81f0c0527e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:03:53.435564 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:53.435406    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv" podUID="a5031ee6-029b-44a4-8c97-68a38a1d12d4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:03:53.925800 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:53.925742    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:54.699304 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:54.699267    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd"]
Oct 30 23:03:54.731839 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:54.731814    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9"]
Oct 30 23:03:54.887129 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:54.887096    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:55.057966 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:55.057930    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:55.228457 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:55.228419    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:55.395299 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:55.395197    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:55.933847 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:55.933810    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:57.132376 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:57.132314    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:57.295516 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:57.295465    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:57.837286 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:57.837218    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:57.875316 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:57.875112    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k" podUID="2de71837-99a5-4e01-ae12-385e722759eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:03:57.875316 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:57.875110    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc" podUID="368f6471-b984-4935-92e6-047c8f81053f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:03:58.030667 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.030602    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv"
Oct 30 23:03:58.056509 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.056482    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5031ee6-029b-44a4-8c97-68a38a1d12d4-kube-api-access-tvn9p\" (UniqueName: \"kubernetes.io/projected/a5031ee6-029b-44a4-8c97-68a38a1d12d4-kube-api-access-tvn9p\") pod \"a5031ee6-029b-44a4-8c97-68a38a1d12d4\" (UID: \"a5031ee6-029b-44a4-8c97-68a38a1d12d4\") "
Oct 30 23:03:58.056657 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.056536    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5031ee6-029b-44a4-8c97-68a38a1d12d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5031ee6-029b-44a4-8c97-68a38a1d12d4-plugins-config-volume\") pod \"a5031ee6-029b-44a4-8c97-68a38a1d12d4\" (UID: \"a5031ee6-029b-44a4-8c97-68a38a1d12d4\") "
Oct 30 23:03:58.058621 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.056897    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5031ee6-029b-44a4-8c97-68a38a1d12d4-plugins-config-volume" pod "a5031ee6-029b-44a4-8c97-68a38a1d12d4" (UID: "a5031ee6-029b-44a4-8c97-68a38a1d12d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:58.058621 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.057602    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5031ee6-029b-44a4-8c97-68a38a1d12d4-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:58.059306 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.059280    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5031ee6-029b-44a4-8c97-68a38a1d12d4-kube-api-access-tvn9p" pod "a5031ee6-029b-44a4-8c97-68a38a1d12d4" (UID: "a5031ee6-029b-44a4-8c97-68a38a1d12d4"). InnerVolumeSpecName "kube-api-access-tvn9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:58.069103 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.069079    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv"
Oct 30 23:03:58.106131 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.106110    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 23:03:58.158257 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.158179    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-credential-socket\") pod \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") "
Oct 30 23:03:58.158733 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.158269    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-workload-certs\") pod \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") "
Oct 30 23:03:58.158733 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.158320    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-kube-api-access-drj8z\" (UniqueName: \"kubernetes.io/projected/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-kube-api-access-drj8z\") pod \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") "
Oct 30 23:03:58.158733 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.158405    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-envoy\") pod \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") "
Oct 30 23:03:58.158733 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.158448    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-workload-socket\") pod \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") "
Oct 30 23:03:58.158733 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.158499    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d131b55f-ea6b-4f31-8053-fe81f0c0527e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d131b55f-ea6b-4f31-8053-fe81f0c0527e-plugins-config-volume\") pod \"d131b55f-ea6b-4f31-8053-fe81f0c0527e\" (UID: \"d131b55f-ea6b-4f31-8053-fe81f0c0527e\") "
Oct 30 23:03:58.158733 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.158535    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-token\" (UniqueName: \"kubernetes.io/projected/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-token\") pod \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") "
Oct 30 23:03:58.158733 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.158603    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istiod-ca-cert\") pod \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") "
Oct 30 23:03:58.158733 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.158623    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-credential-socket" pod "c02298a7-464d-4cc4-8bef-3be1dfc6f3ab" (UID: "c02298a7-464d-4cc4-8bef-3be1dfc6f3ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:58.158733 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.158715    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-data\") pod \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") "
Oct 30 23:03:58.159265 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.158767    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d131b55f-ea6b-4f31-8053-fe81f0c0527e-kube-api-access-lktxs\" (UniqueName: \"kubernetes.io/projected/d131b55f-ea6b-4f31-8053-fe81f0c0527e-kube-api-access-lktxs\") pod \"d131b55f-ea6b-4f31-8053-fe81f0c0527e\" (UID: \"d131b55f-ea6b-4f31-8053-fe81f0c0527e\") "
Oct 30 23:03:58.159265 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.158802    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-podinfo\") pod \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\" (UID: \"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab\") "
Oct 30 23:03:58.159265 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.159006    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-workload-socket" pod "c02298a7-464d-4cc4-8bef-3be1dfc6f3ab" (UID: "c02298a7-464d-4cc4-8bef-3be1dfc6f3ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:58.159265 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.159257    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-data" pod "c02298a7-464d-4cc4-8bef-3be1dfc6f3ab" (UID: "c02298a7-464d-4cc4-8bef-3be1dfc6f3ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:58.159541 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.159331    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:58.159541 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.159353    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:58.159541 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.159370    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvn9p\" (UniqueName: \"kubernetes.io/projected/a5031ee6-029b-44a4-8c97-68a38a1d12d4-kube-api-access-tvn9p\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:58.159730 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.159593    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d131b55f-ea6b-4f31-8053-fe81f0c0527e-plugins-config-volume" pod "d131b55f-ea6b-4f31-8053-fe81f0c0527e" (UID: "d131b55f-ea6b-4f31-8053-fe81f0c0527e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:58.159795 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.159745    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istiod-ca-cert" pod "c02298a7-464d-4cc4-8bef-3be1dfc6f3ab" (UID: "c02298a7-464d-4cc4-8bef-3be1dfc6f3ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:58.159795 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.159768    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-workload-certs" pod "c02298a7-464d-4cc4-8bef-3be1dfc6f3ab" (UID: "c02298a7-464d-4cc4-8bef-3be1dfc6f3ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:58.168673 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.168645    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-podinfo" pod "c02298a7-464d-4cc4-8bef-3be1dfc6f3ab" (UID: "c02298a7-464d-4cc4-8bef-3be1dfc6f3ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:03:58.169833 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.169806    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-kube-api-access-drj8z" pod "c02298a7-464d-4cc4-8bef-3be1dfc6f3ab" (UID: "c02298a7-464d-4cc4-8bef-3be1dfc6f3ab"). InnerVolumeSpecName "kube-api-access-drj8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:58.169976 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.169845    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d131b55f-ea6b-4f31-8053-fe81f0c0527e-kube-api-access-lktxs" pod "d131b55f-ea6b-4f31-8053-fe81f0c0527e" (UID: "d131b55f-ea6b-4f31-8053-fe81f0c0527e"). InnerVolumeSpecName "kube-api-access-lktxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:58.169976 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.169939    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-token" pod "c02298a7-464d-4cc4-8bef-3be1dfc6f3ab" (UID: "c02298a7-464d-4cc4-8bef-3be1dfc6f3ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:58.170109 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.170005    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-envoy" pod "c02298a7-464d-4cc4-8bef-3be1dfc6f3ab" (UID: "c02298a7-464d-4cc4-8bef-3be1dfc6f3ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:58.261019 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.260988    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:58.261019 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.261015    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lktxs\" (UniqueName: \"kubernetes.io/projected/d131b55f-ea6b-4f31-8053-fe81f0c0527e-kube-api-access-lktxs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:58.261019 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.261027    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:58.261263 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.261036    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:58.261263 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.261045    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drj8z\" (UniqueName: \"kubernetes.io/projected/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-kube-api-access-drj8z\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:58.261263 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.261060    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:58.261263 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.261073    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d131b55f-ea6b-4f31-8053-fe81f0c0527e-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:58.261263 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.261082    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:58.261263 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.261090    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:03:58.518343 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.518304    4889 generic.go:356] "Generic (PLEG): container finished" podID="a5031ee6-029b-44a4-8c97-68a38a1d12d4" containerID="d1868142ecb0b03833ecdd7ea9169196cbf8a5f116317a138bc986035cbabda9" exitCode=0
Oct 30 23:03:58.518536 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.518354    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv"
Oct 30 23:03:58.518536 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.518385    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv" event={"ID":"a5031ee6-029b-44a4-8c97-68a38a1d12d4","Type":"ContainerDied","Data":"d1868142ecb0b03833ecdd7ea9169196cbf8a5f116317a138bc986035cbabda9"}
Oct 30 23:03:58.518536 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.518439    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv" event={"ID":"a5031ee6-029b-44a4-8c97-68a38a1d12d4","Type":"ContainerDied","Data":"31f4ccec5e636713a4d6be83b68cfa83405fbe2ff18facd263ab63da45983a2a"}
Oct 30 23:03:58.518536 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.518465    4889 scope.go:122] "RemoveContainer" containerID="d1868142ecb0b03833ecdd7ea9169196cbf8a5f116317a138bc986035cbabda9"
Oct 30 23:03:58.520563 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.520541    4889 generic.go:356] "Generic (PLEG): container finished" podID="c02298a7-464d-4cc4-8bef-3be1dfc6f3ab" containerID="be3cd57e007b64c0705a95159a6611725bdd672c86bd63fb3d1e9bc05d1cfc3b" exitCode=0
Oct 30 23:03:58.520707 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.520590    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"
Oct 30 23:03:58.520707 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.520638    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg" event={"ID":"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab","Type":"ContainerDied","Data":"be3cd57e007b64c0705a95159a6611725bdd672c86bd63fb3d1e9bc05d1cfc3b"}
Oct 30 23:03:58.520707 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.520695    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg" event={"ID":"c02298a7-464d-4cc4-8bef-3be1dfc6f3ab","Type":"ContainerDied","Data":"01d8938e3ee17f4d72333e7c5c4fb075fb040ad250d91c03889304a9633ad398"}
Oct 30 23:03:58.523860 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.523835    4889 generic.go:356] "Generic (PLEG): container finished" podID="d131b55f-ea6b-4f31-8053-fe81f0c0527e" containerID="1d8c057bbf32c5644bf5f293ee973b4ec7c0609ef55dfd177698287d52a94534" exitCode=0
Oct 30 23:03:58.523962 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.523879    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv"
Oct 30 23:03:58.523962 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.523902    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv" event={"ID":"d131b55f-ea6b-4f31-8053-fe81f0c0527e","Type":"ContainerDied","Data":"1d8c057bbf32c5644bf5f293ee973b4ec7c0609ef55dfd177698287d52a94534"}
Oct 30 23:03:58.523962 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.523944    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv" event={"ID":"d131b55f-ea6b-4f31-8053-fe81f0c0527e","Type":"ContainerDied","Data":"4fd8a64bd322051827a264ac39aeb4fda4630ffc838fc4c0d894cefb981db4d2"}
Oct 30 23:03:58.531377 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.531352    4889 scope.go:122] "RemoveContainer" containerID="d1868142ecb0b03833ecdd7ea9169196cbf8a5f116317a138bc986035cbabda9"
Oct 30 23:03:58.531813 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:03:58.531784    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1868142ecb0b03833ecdd7ea9169196cbf8a5f116317a138bc986035cbabda9\": container with ID starting with d1868142ecb0b03833ecdd7ea9169196cbf8a5f116317a138bc986035cbabda9 not found: ID does not exist" containerID="d1868142ecb0b03833ecdd7ea9169196cbf8a5f116317a138bc986035cbabda9"
Oct 30 23:03:58.531881 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.531823    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1868142ecb0b03833ecdd7ea9169196cbf8a5f116317a138bc986035cbabda9"} err="failed to get container status \"d1868142ecb0b03833ecdd7ea9169196cbf8a5f116317a138bc986035cbabda9\": rpc error: code = NotFound desc = could not find container \"d1868142ecb0b03833ecdd7ea9169196cbf8a5f116317a138bc986035cbabda9\": container with ID starting with d1868142ecb0b03833ecdd7ea9169196cbf8a5f116317a138bc986035cbabda9 not found: ID does not exist"
Oct 30 23:03:58.531881 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.531845    4889 scope.go:122] "RemoveContainer" containerID="be3cd57e007b64c0705a95159a6611725bdd672c86bd63fb3d1e9bc05d1cfc3b"
Oct 30 23:03:58.542375 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.542359    4889 scope.go:122] "RemoveContainer" containerID="acee3959f7a8ff1c1b5e6f4e56a3c97046c0955298ab9883b0a770de90546c16"
Oct 30 23:03:58.551461 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.551444    4889 scope.go:122] "RemoveContainer" containerID="be3cd57e007b64c0705a95159a6611725bdd672c86bd63fb3d1e9bc05d1cfc3b"
Oct 30 23:03:58.551724 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:03:58.551705    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be3cd57e007b64c0705a95159a6611725bdd672c86bd63fb3d1e9bc05d1cfc3b\": container with ID starting with be3cd57e007b64c0705a95159a6611725bdd672c86bd63fb3d1e9bc05d1cfc3b not found: ID does not exist" containerID="be3cd57e007b64c0705a95159a6611725bdd672c86bd63fb3d1e9bc05d1cfc3b"
Oct 30 23:03:58.551797 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.551731    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be3cd57e007b64c0705a95159a6611725bdd672c86bd63fb3d1e9bc05d1cfc3b"} err="failed to get container status \"be3cd57e007b64c0705a95159a6611725bdd672c86bd63fb3d1e9bc05d1cfc3b\": rpc error: code = NotFound desc = could not find container \"be3cd57e007b64c0705a95159a6611725bdd672c86bd63fb3d1e9bc05d1cfc3b\": container with ID starting with be3cd57e007b64c0705a95159a6611725bdd672c86bd63fb3d1e9bc05d1cfc3b not found: ID does not exist"
Oct 30 23:03:58.551797 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.551749    4889 scope.go:122] "RemoveContainer" containerID="acee3959f7a8ff1c1b5e6f4e56a3c97046c0955298ab9883b0a770de90546c16"
Oct 30 23:03:58.551993 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:03:58.551973    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acee3959f7a8ff1c1b5e6f4e56a3c97046c0955298ab9883b0a770de90546c16\": container with ID starting with acee3959f7a8ff1c1b5e6f4e56a3c97046c0955298ab9883b0a770de90546c16 not found: ID does not exist" containerID="acee3959f7a8ff1c1b5e6f4e56a3c97046c0955298ab9883b0a770de90546c16"
Oct 30 23:03:58.552038 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.552001    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acee3959f7a8ff1c1b5e6f4e56a3c97046c0955298ab9883b0a770de90546c16"} err="failed to get container status \"acee3959f7a8ff1c1b5e6f4e56a3c97046c0955298ab9883b0a770de90546c16\": rpc error: code = NotFound desc = could not find container \"acee3959f7a8ff1c1b5e6f4e56a3c97046c0955298ab9883b0a770de90546c16\": container with ID starting with acee3959f7a8ff1c1b5e6f4e56a3c97046c0955298ab9883b0a770de90546c16 not found: ID does not exist"
Oct 30 23:03:58.552038 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.552017    4889 scope.go:122] "RemoveContainer" containerID="1d8c057bbf32c5644bf5f293ee973b4ec7c0609ef55dfd177698287d52a94534"
Oct 30 23:03:58.561473 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.561456    4889 scope.go:122] "RemoveContainer" containerID="1d8c057bbf32c5644bf5f293ee973b4ec7c0609ef55dfd177698287d52a94534"
Oct 30 23:03:58.561769 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:03:58.561751    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8c057bbf32c5644bf5f293ee973b4ec7c0609ef55dfd177698287d52a94534\": container with ID starting with 1d8c057bbf32c5644bf5f293ee973b4ec7c0609ef55dfd177698287d52a94534 not found: ID does not exist" containerID="1d8c057bbf32c5644bf5f293ee973b4ec7c0609ef55dfd177698287d52a94534"
Oct 30 23:03:58.561834 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.561775    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8c057bbf32c5644bf5f293ee973b4ec7c0609ef55dfd177698287d52a94534"} err="failed to get container status \"1d8c057bbf32c5644bf5f293ee973b4ec7c0609ef55dfd177698287d52a94534\": rpc error: code = NotFound desc = could not find container \"1d8c057bbf32c5644bf5f293ee973b4ec7c0609ef55dfd177698287d52a94534\": container with ID starting with 1d8c057bbf32c5644bf5f293ee973b4ec7c0609ef55dfd177698287d52a94534 not found: ID does not exist"
Oct 30 23:03:58.597949 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.597921    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"]
Oct 30 23:03:58.602048 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.602019    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-bbqjg"]
Oct 30 23:03:58.639235 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.639197    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv"]
Oct 30 23:03:58.645614 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.645590    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bed544-5b5dc64c47-5dwdv"]
Oct 30 23:03:58.682428 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.682391    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv"]
Oct 30 23:03:58.688132 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.688102    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bed544-6bc95f58bc-xmfrv"]
Oct 30 23:03:58.782795 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.782719    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5031ee6-029b-44a4-8c97-68a38a1d12d4" path="/var/lib/kubelet/pods/a5031ee6-029b-44a4-8c97-68a38a1d12d4/volumes"
Oct 30 23:03:58.783265 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.783248    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c02298a7-464d-4cc4-8bef-3be1dfc6f3ab" path="/var/lib/kubelet/pods/c02298a7-464d-4cc4-8bef-3be1dfc6f3ab/volumes"
Oct 30 23:03:58.783663 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:58.783649    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d131b55f-ea6b-4f31-8053-fe81f0c0527e" path="/var/lib/kubelet/pods/d131b55f-ea6b-4f31-8053-fe81f0c0527e/volumes"
Oct 30 23:03:59.516727 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:59.516691    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:59.702128 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:59.700364    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd" podUID="341cd65b-6e79-41b5-996c-1b866e57780c" containerName="payload-processing" containerID="cri-o://19a2d32ad49e78a0353850ac47e9854be819d22c5cf249b26249a4cdef9eeeaa" gracePeriod=25
Oct 30 23:03:59.732312 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:59.732266    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9" podUID="b37648ec-d503-43af-aef6-ec9da60a0902" containerName="payload-pre-processing" containerID="cri-o://0b0e660e33109897d30c5f9f6d29a88f0f7a8b700186279151228cccb1533f3d" gracePeriod=25
Oct 30 23:03:59.750183 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:03:59.750157    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:01.404817 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:01.404779    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:01.603570 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:01.603529    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9" podUID="b37648ec-d503-43af-aef6-ec9da60a0902" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:04:02.624206 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:02.624163    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd" podUID="341cd65b-6e79-41b5-996c-1b866e57780c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:04:03.350331 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:03.350296    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:03.819640 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:03.819601    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b"]
Oct 30 23:04:03.999938 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:03.999903    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:04.171055 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:04.170960    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:04.567834 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:04.567804    4889 generic.go:356] "Generic (PLEG): container finished" podID="368f6471-b984-4935-92e6-047c8f81053f" containerID="bb25d1a297429fc084984b461e011fcaa9b45b2995679e8d2e475ec02fe88956" exitCode=0
Oct 30 23:04:04.568015 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:04.567879    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc" event={"ID":"368f6471-b984-4935-92e6-047c8f81053f","Type":"ContainerDied","Data":"bb25d1a297429fc084984b461e011fcaa9b45b2995679e8d2e475ec02fe88956"}
Oct 30 23:04:04.569572 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:04.569548    4889 generic.go:356] "Generic (PLEG): container finished" podID="2de71837-99a5-4e01-ae12-385e722759eb" containerID="f9de8cfaec16dc8ce77ddc7fd433efc4209f7a53eed48341dd0befa46505c4b5" exitCode=0
Oct 30 23:04:04.569710 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:04.569625    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k" event={"ID":"2de71837-99a5-4e01-ae12-385e722759eb","Type":"ContainerDied","Data":"f9de8cfaec16dc8ce77ddc7fd433efc4209f7a53eed48341dd0befa46505c4b5"}
Oct 30 23:04:04.613706 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:04.613677    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc"
Oct 30 23:04:04.641238 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:04.641199    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k"
Oct 30 23:04:04.732711 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:04.732610    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2de71837-99a5-4e01-ae12-385e722759eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de71837-99a5-4e01-ae12-385e722759eb-plugins-config-volume\") pod \"2de71837-99a5-4e01-ae12-385e722759eb\" (UID: \"2de71837-99a5-4e01-ae12-385e722759eb\") "
Oct 30 23:04:04.732711 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:04.732667    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/368f6471-b984-4935-92e6-047c8f81053f-kube-api-access-h798f\" (UniqueName: \"kubernetes.io/projected/368f6471-b984-4935-92e6-047c8f81053f-kube-api-access-h798f\") pod \"368f6471-b984-4935-92e6-047c8f81053f\" (UID: \"368f6471-b984-4935-92e6-047c8f81053f\") "
Oct 30 23:04:04.732956 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:04.732715    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/368f6471-b984-4935-92e6-047c8f81053f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/368f6471-b984-4935-92e6-047c8f81053f-plugins-config-volume\") pod \"368f6471-b984-4935-92e6-047c8f81053f\" (UID: \"368f6471-b984-4935-92e6-047c8f81053f\") "
Oct 30 23:04:04.732956 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:04.732858    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2de71837-99a5-4e01-ae12-385e722759eb-kube-api-access-chrhb\" (UniqueName: \"kubernetes.io/projected/2de71837-99a5-4e01-ae12-385e722759eb-kube-api-access-chrhb\") pod \"2de71837-99a5-4e01-ae12-385e722759eb\" (UID: \"2de71837-99a5-4e01-ae12-385e722759eb\") "
Oct 30 23:04:04.733116 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:04.733085    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2de71837-99a5-4e01-ae12-385e722759eb-plugins-config-volume" pod "2de71837-99a5-4e01-ae12-385e722759eb" (UID: "2de71837-99a5-4e01-ae12-385e722759eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:04.733297 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:04.733114    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/368f6471-b984-4935-92e6-047c8f81053f-plugins-config-volume" pod "368f6471-b984-4935-92e6-047c8f81053f" (UID: "368f6471-b984-4935-92e6-047c8f81053f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:04.733491 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:04.733471    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de71837-99a5-4e01-ae12-385e722759eb-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:04.733553 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:04.733496    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/368f6471-b984-4935-92e6-047c8f81053f-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:04.735011 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:04.734989    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/368f6471-b984-4935-92e6-047c8f81053f-kube-api-access-h798f" pod "368f6471-b984-4935-92e6-047c8f81053f" (UID: "368f6471-b984-4935-92e6-047c8f81053f"). InnerVolumeSpecName "kube-api-access-h798f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:04.735216 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:04.735198    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de71837-99a5-4e01-ae12-385e722759eb-kube-api-access-chrhb" pod "2de71837-99a5-4e01-ae12-385e722759eb" (UID: "2de71837-99a5-4e01-ae12-385e722759eb"). InnerVolumeSpecName "kube-api-access-chrhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:04.835393 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:04.835364    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chrhb\" (UniqueName: \"kubernetes.io/projected/2de71837-99a5-4e01-ae12-385e722759eb-kube-api-access-chrhb\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:04.835393 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:04.835391    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h798f\" (UniqueName: \"kubernetes.io/projected/368f6471-b984-4935-92e6-047c8f81053f-kube-api-access-h798f\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:05.578342 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:05.578237    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc" event={"ID":"368f6471-b984-4935-92e6-047c8f81053f","Type":"ContainerDied","Data":"04d25166b2a14f36dac24eab3dde67574110b19ec02c047c1eb475e37b1606b3"}
Oct 30 23:04:05.578342 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:05.578264    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc"
Oct 30 23:04:05.578342 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:05.578332    4889 scope.go:122] "RemoveContainer" containerID="bb25d1a297429fc084984b461e011fcaa9b45b2995679e8d2e475ec02fe88956"
Oct 30 23:04:05.580011 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:05.579987    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k"
Oct 30 23:04:05.580178 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:05.579984    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k" event={"ID":"2de71837-99a5-4e01-ae12-385e722759eb","Type":"ContainerDied","Data":"8233408b8fb3d926fc9c7bafe57ccd2da99600594233b519fac3d0746793583b"}
Oct 30 23:04:05.590959 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:05.590935    4889 scope.go:122] "RemoveContainer" containerID="f9de8cfaec16dc8ce77ddc7fd433efc4209f7a53eed48341dd0befa46505c4b5"
Oct 30 23:04:05.634942 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:05.634911    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k"]
Oct 30 23:04:05.638444 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:05.638418    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5983ea-f76b4bb8c-xvp6k"]
Oct 30 23:04:05.688492 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:05.688446    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc"]
Oct 30 23:04:05.692051 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:05.692021    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5983ea-65fbff47dc-kjmcc"]
Oct 30 23:04:05.708172 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:05.708146    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:05.904655 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:05.904561    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:06.082550 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:06.082508    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:06.511487 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:06.511453    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:06.722478 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:06.722445    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:06.779184 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:06.779103    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2de71837-99a5-4e01-ae12-385e722759eb" path="/var/lib/kubelet/pods/2de71837-99a5-4e01-ae12-385e722759eb/volumes"
Oct 30 23:04:06.779755 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:06.779732    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="368f6471-b984-4935-92e6-047c8f81053f" path="/var/lib/kubelet/pods/368f6471-b984-4935-92e6-047c8f81053f/volumes"
Oct 30 23:04:07.218403 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:07.218279    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:07.734768 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:07.734730    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:07.912074 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:07.912037    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:08.325105 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:08.325069    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:08.820202 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:08.820161    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b" podUID="dafe75fd-77ad-49d3-aafb-35f28fcb2954" containerName="payload-pre-processing" containerID="cri-o://21096e1f1e587ecd0afcb91eaefabb23562d761057706773c44598c9a8484c57" gracePeriod=25
Oct 30 23:04:09.110387 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:09.110363    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b"
Oct 30 23:04:09.201208 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:09.201171    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dafe75fd-77ad-49d3-aafb-35f28fcb2954-kube-api-access-bh8d2\" (UniqueName: \"kubernetes.io/projected/dafe75fd-77ad-49d3-aafb-35f28fcb2954-kube-api-access-bh8d2\") pod \"dafe75fd-77ad-49d3-aafb-35f28fcb2954\" (UID: \"dafe75fd-77ad-49d3-aafb-35f28fcb2954\") "
Oct 30 23:04:09.201422 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:09.201301    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dafe75fd-77ad-49d3-aafb-35f28fcb2954-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dafe75fd-77ad-49d3-aafb-35f28fcb2954-plugins-config-volume\") pod \"dafe75fd-77ad-49d3-aafb-35f28fcb2954\" (UID: \"dafe75fd-77ad-49d3-aafb-35f28fcb2954\") "
Oct 30 23:04:09.201780 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:09.201745    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dafe75fd-77ad-49d3-aafb-35f28fcb2954-plugins-config-volume" pod "dafe75fd-77ad-49d3-aafb-35f28fcb2954" (UID: "dafe75fd-77ad-49d3-aafb-35f28fcb2954"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:09.204052 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:09.204027    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafe75fd-77ad-49d3-aafb-35f28fcb2954-kube-api-access-bh8d2" pod "dafe75fd-77ad-49d3-aafb-35f28fcb2954" (UID: "dafe75fd-77ad-49d3-aafb-35f28fcb2954"). InnerVolumeSpecName "kube-api-access-bh8d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:09.303278 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:09.303249    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh8d2\" (UniqueName: \"kubernetes.io/projected/dafe75fd-77ad-49d3-aafb-35f28fcb2954-kube-api-access-bh8d2\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:09.303278 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:09.303276    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dafe75fd-77ad-49d3-aafb-35f28fcb2954-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:09.613088 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:09.613049    4889 generic.go:356] "Generic (PLEG): container finished" podID="dafe75fd-77ad-49d3-aafb-35f28fcb2954" containerID="21096e1f1e587ecd0afcb91eaefabb23562d761057706773c44598c9a8484c57" exitCode=0
Oct 30 23:04:09.613088 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:09.613084    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b"
Oct 30 23:04:09.613648 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:09.613120    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b" event={"ID":"dafe75fd-77ad-49d3-aafb-35f28fcb2954","Type":"ContainerDied","Data":"21096e1f1e587ecd0afcb91eaefabb23562d761057706773c44598c9a8484c57"}
Oct 30 23:04:09.613648 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:09.613172    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b" event={"ID":"dafe75fd-77ad-49d3-aafb-35f28fcb2954","Type":"ContainerDied","Data":"ffef39e2aab3f7ddcf1140535d7f36ac30e6c5e5c79e9f1aad96976dd59f9da2"}
Oct 30 23:04:09.613648 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:09.613200    4889 scope.go:122] "RemoveContainer" containerID="21096e1f1e587ecd0afcb91eaefabb23562d761057706773c44598c9a8484c57"
Oct 30 23:04:09.624114 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:09.624090    4889 scope.go:122] "RemoveContainer" containerID="21096e1f1e587ecd0afcb91eaefabb23562d761057706773c44598c9a8484c57"
Oct 30 23:04:09.624440 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:04:09.624418    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21096e1f1e587ecd0afcb91eaefabb23562d761057706773c44598c9a8484c57\": container with ID starting with 21096e1f1e587ecd0afcb91eaefabb23562d761057706773c44598c9a8484c57 not found: ID does not exist" containerID="21096e1f1e587ecd0afcb91eaefabb23562d761057706773c44598c9a8484c57"
Oct 30 23:04:09.624519 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:09.624448    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21096e1f1e587ecd0afcb91eaefabb23562d761057706773c44598c9a8484c57"} err="failed to get container status \"21096e1f1e587ecd0afcb91eaefabb23562d761057706773c44598c9a8484c57\": rpc error: code = NotFound desc = could not find container \"21096e1f1e587ecd0afcb91eaefabb23562d761057706773c44598c9a8484c57\": container with ID starting with 21096e1f1e587ecd0afcb91eaefabb23562d761057706773c44598c9a8484c57 not found: ID does not exist"
Oct 30 23:04:09.667504 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:09.667468    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b"]
Oct 30 23:04:09.680880 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:09.680843    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6af86e-64944b4ff9-4229b"]
Oct 30 23:04:09.851736 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:09.851702    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:10.302759 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:04:10.302716    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:04:10.774386 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:10.774354    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dafe75fd-77ad-49d3-aafb-35f28fcb2954" path="/var/lib/kubelet/pods/dafe75fd-77ad-49d3-aafb-35f28fcb2954/volumes"
Oct 30 23:04:11.382550 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:11.382509    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:11.603406 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:11.603364    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9" podUID="b37648ec-d503-43af-aef6-ec9da60a0902" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:04:11.659417 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:11.659336    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf"]
Oct 30 23:04:11.659622 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:11.659595    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf" podUID="664cb2dc-09fb-45ca-b315-7ab915110da7" containerName="maas-api" containerID="cri-o://ac9513720d3f26a79085d38105323e9a3df48c23cf735f860b99ef51bde811e5" gracePeriod=30
Oct 30 23:04:11.781933 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:11.781898    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:11.820363 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:11.820330    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2"]
Oct 30 23:04:11.820554 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:11.820533    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2" podUID="428b38e7-52ac-4f32-a8be-ea74f109c4d1" containerName="payload-processing" containerID="cri-o://8d9d49f2df099fdd27f0e79993b61396725a2b85157663cff4e9991f7eebab72" gracePeriod=30
Oct 30 23:04:11.953922 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:11.945007    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:11.989858 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:11.989830    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf"
Oct 30 23:04:12.093472 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.093449    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2"
Oct 30 23:04:12.111770 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.111742    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:12.138038 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.137990    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/664cb2dc-09fb-45ca-b315-7ab915110da7-kube-api-access-9n27f\" (UniqueName: \"kubernetes.io/projected/664cb2dc-09fb-45ca-b315-7ab915110da7-kube-api-access-9n27f\") pod \"664cb2dc-09fb-45ca-b315-7ab915110da7\" (UID: \"664cb2dc-09fb-45ca-b315-7ab915110da7\") "
Oct 30 23:04:12.138348 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.138155    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/664cb2dc-09fb-45ca-b315-7ab915110da7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/664cb2dc-09fb-45ca-b315-7ab915110da7-openshift-service-ca\") pod \"664cb2dc-09fb-45ca-b315-7ab915110da7\" (UID: \"664cb2dc-09fb-45ca-b315-7ab915110da7\") "
Oct 30 23:04:12.138348 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.138280    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/664cb2dc-09fb-45ca-b315-7ab915110da7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/664cb2dc-09fb-45ca-b315-7ab915110da7-metrics-certs\") pod \"664cb2dc-09fb-45ca-b315-7ab915110da7\" (UID: \"664cb2dc-09fb-45ca-b315-7ab915110da7\") "
Oct 30 23:04:12.138621 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.138400    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/664cb2dc-09fb-45ca-b315-7ab915110da7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/664cb2dc-09fb-45ca-b315-7ab915110da7-maas-api-tls\") pod \"664cb2dc-09fb-45ca-b315-7ab915110da7\" (UID: \"664cb2dc-09fb-45ca-b315-7ab915110da7\") "
Oct 30 23:04:12.138704 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.138655    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/664cb2dc-09fb-45ca-b315-7ab915110da7-openshift-service-ca" pod "664cb2dc-09fb-45ca-b315-7ab915110da7" (UID: "664cb2dc-09fb-45ca-b315-7ab915110da7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:12.138910 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.138876    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/664cb2dc-09fb-45ca-b315-7ab915110da7-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:12.141829 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.141217    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/664cb2dc-09fb-45ca-b315-7ab915110da7-kube-api-access-9n27f" pod "664cb2dc-09fb-45ca-b315-7ab915110da7" (UID: "664cb2dc-09fb-45ca-b315-7ab915110da7"). InnerVolumeSpecName "kube-api-access-9n27f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:12.141951 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.141827    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/664cb2dc-09fb-45ca-b315-7ab915110da7-maas-api-tls" pod "664cb2dc-09fb-45ca-b315-7ab915110da7" (UID: "664cb2dc-09fb-45ca-b315-7ab915110da7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:12.141951 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.141928    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/664cb2dc-09fb-45ca-b315-7ab915110da7-metrics-certs" pod "664cb2dc-09fb-45ca-b315-7ab915110da7" (UID: "664cb2dc-09fb-45ca-b315-7ab915110da7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:12.239899 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.239812    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/428b38e7-52ac-4f32-a8be-ea74f109c4d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428b38e7-52ac-4f32-a8be-ea74f109c4d1-plugins-config-volume\") pod \"428b38e7-52ac-4f32-a8be-ea74f109c4d1\" (UID: \"428b38e7-52ac-4f32-a8be-ea74f109c4d1\") "
Oct 30 23:04:12.240102 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.239957    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/428b38e7-52ac-4f32-a8be-ea74f109c4d1-kube-api-access-qmg42\" (UniqueName: \"kubernetes.io/projected/428b38e7-52ac-4f32-a8be-ea74f109c4d1-kube-api-access-qmg42\") pod \"428b38e7-52ac-4f32-a8be-ea74f109c4d1\" (UID: \"428b38e7-52ac-4f32-a8be-ea74f109c4d1\") "
Oct 30 23:04:12.240313 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.240286    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428b38e7-52ac-4f32-a8be-ea74f109c4d1-plugins-config-volume" pod "428b38e7-52ac-4f32-a8be-ea74f109c4d1" (UID: "428b38e7-52ac-4f32-a8be-ea74f109c4d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:12.240528 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.240459    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/664cb2dc-09fb-45ca-b315-7ab915110da7-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:12.240528 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.240484    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n27f\" (UniqueName: \"kubernetes.io/projected/664cb2dc-09fb-45ca-b315-7ab915110da7-kube-api-access-9n27f\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:12.240528 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.240496    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428b38e7-52ac-4f32-a8be-ea74f109c4d1-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:12.240528 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.240505    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/664cb2dc-09fb-45ca-b315-7ab915110da7-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:12.246431 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.246404    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428b38e7-52ac-4f32-a8be-ea74f109c4d1-kube-api-access-qmg42" pod "428b38e7-52ac-4f32-a8be-ea74f109c4d1" (UID: "428b38e7-52ac-4f32-a8be-ea74f109c4d1"). InnerVolumeSpecName "kube-api-access-qmg42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:12.341854 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.341822    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmg42\" (UniqueName: \"kubernetes.io/projected/428b38e7-52ac-4f32-a8be-ea74f109c4d1-kube-api-access-qmg42\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:12.488913 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.488860    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:12.624368 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.624328    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd" podUID="341cd65b-6e79-41b5-996c-1b866e57780c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:04:12.636262 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.634653    4889 generic.go:356] "Generic (PLEG): container finished" podID="428b38e7-52ac-4f32-a8be-ea74f109c4d1" containerID="8d9d49f2df099fdd27f0e79993b61396725a2b85157663cff4e9991f7eebab72" exitCode=0
Oct 30 23:04:12.636262 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.634761    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2" event={"ID":"428b38e7-52ac-4f32-a8be-ea74f109c4d1","Type":"ContainerDied","Data":"8d9d49f2df099fdd27f0e79993b61396725a2b85157663cff4e9991f7eebab72"}
Oct 30 23:04:12.642045 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.641980    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2" event={"ID":"428b38e7-52ac-4f32-a8be-ea74f109c4d1","Type":"ContainerDied","Data":"66abc1da68bbb22ad5b8cc4f4a426c4657add692ca6a640b7eda0d03e51babaa"}
Oct 30 23:04:12.642045 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.642045    4889 scope.go:122] "RemoveContainer" containerID="8d9d49f2df099fdd27f0e79993b61396725a2b85157663cff4e9991f7eebab72"
Oct 30 23:04:12.642451 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.642409    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2"
Oct 30 23:04:12.645259 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.645210    4889 generic.go:356] "Generic (PLEG): container finished" podID="664cb2dc-09fb-45ca-b315-7ab915110da7" containerID="ac9513720d3f26a79085d38105323e9a3df48c23cf735f860b99ef51bde811e5" exitCode=0
Oct 30 23:04:12.645416 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.645375    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf"
Oct 30 23:04:12.645579 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.645435    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf" event={"ID":"664cb2dc-09fb-45ca-b315-7ab915110da7","Type":"ContainerDied","Data":"ac9513720d3f26a79085d38105323e9a3df48c23cf735f860b99ef51bde811e5"}
Oct 30 23:04:12.645732 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.645623    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf" event={"ID":"664cb2dc-09fb-45ca-b315-7ab915110da7","Type":"ContainerDied","Data":"85e5be789b867f1a6eefdcce88e7f6dccd5a8810331498bb609821d75dc83d82"}
Oct 30 23:04:12.654396 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.654363    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:12.655356 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.655336    4889 scope.go:122] "RemoveContainer" containerID="8d9d49f2df099fdd27f0e79993b61396725a2b85157663cff4e9991f7eebab72"
Oct 30 23:04:12.655779 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:04:12.655746    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d9d49f2df099fdd27f0e79993b61396725a2b85157663cff4e9991f7eebab72\": container with ID starting with 8d9d49f2df099fdd27f0e79993b61396725a2b85157663cff4e9991f7eebab72 not found: ID does not exist" containerID="8d9d49f2df099fdd27f0e79993b61396725a2b85157663cff4e9991f7eebab72"
Oct 30 23:04:12.655887 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.655787    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d9d49f2df099fdd27f0e79993b61396725a2b85157663cff4e9991f7eebab72"} err="failed to get container status \"8d9d49f2df099fdd27f0e79993b61396725a2b85157663cff4e9991f7eebab72\": rpc error: code = NotFound desc = could not find container \"8d9d49f2df099fdd27f0e79993b61396725a2b85157663cff4e9991f7eebab72\": container with ID starting with 8d9d49f2df099fdd27f0e79993b61396725a2b85157663cff4e9991f7eebab72 not found: ID does not exist"
Oct 30 23:04:12.655887 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.655804    4889 scope.go:122] "RemoveContainer" containerID="ac9513720d3f26a79085d38105323e9a3df48c23cf735f860b99ef51bde811e5"
Oct 30 23:04:12.665203 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.665049    4889 scope.go:122] "RemoveContainer" containerID="ac9513720d3f26a79085d38105323e9a3df48c23cf735f860b99ef51bde811e5"
Oct 30 23:04:12.665381 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:04:12.665360    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac9513720d3f26a79085d38105323e9a3df48c23cf735f860b99ef51bde811e5\": container with ID starting with ac9513720d3f26a79085d38105323e9a3df48c23cf735f860b99ef51bde811e5 not found: ID does not exist" containerID="ac9513720d3f26a79085d38105323e9a3df48c23cf735f860b99ef51bde811e5"
Oct 30 23:04:12.665444 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.665394    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac9513720d3f26a79085d38105323e9a3df48c23cf735f860b99ef51bde811e5"} err="failed to get container status \"ac9513720d3f26a79085d38105323e9a3df48c23cf735f860b99ef51bde811e5\": rpc error: code = NotFound desc = could not find container \"ac9513720d3f26a79085d38105323e9a3df48c23cf735f860b99ef51bde811e5\": container with ID starting with ac9513720d3f26a79085d38105323e9a3df48c23cf735f860b99ef51bde811e5 not found: ID does not exist"
Oct 30 23:04:12.701774 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.701741    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf"]
Oct 30 23:04:12.715272 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.715248    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f94561-b4647f5dc-8fmnf"]
Oct 30 23:04:12.753877 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.753842    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2"]
Oct 30 23:04:12.759443 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.759421    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f94561-b76dcb6f6-jvts2"]
Oct 30 23:04:12.774380 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.774355    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="428b38e7-52ac-4f32-a8be-ea74f109c4d1" path="/var/lib/kubelet/pods/428b38e7-52ac-4f32-a8be-ea74f109c4d1/volumes"
Oct 30 23:04:12.774845 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:12.774829    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="664cb2dc-09fb-45ca-b315-7ab915110da7" path="/var/lib/kubelet/pods/664cb2dc-09fb-45ca-b315-7ab915110da7/volumes"
Oct 30 23:04:13.209538 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:13.209506    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:14.437535 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:14.437501    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:15.383154 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:15.383118    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"]
Oct 30 23:04:15.383450 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:15.383363    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x" podUID="24e8fc0c-97b0-47f1-ba4b-5ec430718d31" containerName="istio-proxy" containerID="cri-o://533f76e65b54feab7f5111481ed774d33442f3ac754e6eddc48347cda2f0e85b" gracePeriod=30
Oct 30 23:04:15.673333 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:15.673212    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:15.863163 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:15.862853    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:16.069615 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:16.069581    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:16.071383 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:16.071354    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"]
Oct 30 23:04:16.071607 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:16.071585    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp" podUID="65b844a4-133d-40a6-bc49-81bbffc94221" containerName="istio-proxy" containerID="cri-o://6cbde591fc250ce6ef68b813f77a7c02533d36d7bb56a89dd70ce9992f5eedc8" gracePeriod=30
Oct 30 23:04:16.278247 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:16.271982    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:16.460185 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:16.460087    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:16.680839 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:16.680791    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:17.158766 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:17.158728    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:17.345701 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:17.345655    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:17.662618 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:17.662582    4889 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:04:17.662817 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:17.662645    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp" podUID="65b844a4-133d-40a6-bc49-81bbffc94221" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:04:17.912471 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:17.912438    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:19.191112 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:19.191069    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:19.820656 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:19.820611    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:20.006151 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.005924    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:20.013815 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.013790    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd"
Oct 30 23:04:20.036727 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.036693    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9"
Oct 30 23:04:20.135025 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.134991    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/341cd65b-6e79-41b5-996c-1b866e57780c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/341cd65b-6e79-41b5-996c-1b866e57780c-plugins-config-volume\") pod \"341cd65b-6e79-41b5-996c-1b866e57780c\" (UID: \"341cd65b-6e79-41b5-996c-1b866e57780c\") "
Oct 30 23:04:20.135215 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.135193    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b37648ec-d503-43af-aef6-ec9da60a0902-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37648ec-d503-43af-aef6-ec9da60a0902-plugins-config-volume\") pod \"b37648ec-d503-43af-aef6-ec9da60a0902\" (UID: \"b37648ec-d503-43af-aef6-ec9da60a0902\") "
Oct 30 23:04:20.135434 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.135408    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/341cd65b-6e79-41b5-996c-1b866e57780c-kube-api-access-b6kcb\" (UniqueName: \"kubernetes.io/projected/341cd65b-6e79-41b5-996c-1b866e57780c-kube-api-access-b6kcb\") pod \"341cd65b-6e79-41b5-996c-1b866e57780c\" (UID: \"341cd65b-6e79-41b5-996c-1b866e57780c\") "
Oct 30 23:04:20.135514 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.135433    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/341cd65b-6e79-41b5-996c-1b866e57780c-plugins-config-volume" pod "341cd65b-6e79-41b5-996c-1b866e57780c" (UID: "341cd65b-6e79-41b5-996c-1b866e57780c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:20.135584 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.135519    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b37648ec-d503-43af-aef6-ec9da60a0902-kube-api-access-8rh9l\" (UniqueName: \"kubernetes.io/projected/b37648ec-d503-43af-aef6-ec9da60a0902-kube-api-access-8rh9l\") pod \"b37648ec-d503-43af-aef6-ec9da60a0902\" (UID: \"b37648ec-d503-43af-aef6-ec9da60a0902\") "
Oct 30 23:04:20.135689 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.135667    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b37648ec-d503-43af-aef6-ec9da60a0902-plugins-config-volume" pod "b37648ec-d503-43af-aef6-ec9da60a0902" (UID: "b37648ec-d503-43af-aef6-ec9da60a0902"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:20.136040 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.136015    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/341cd65b-6e79-41b5-996c-1b866e57780c-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:20.136325 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.136046    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37648ec-d503-43af-aef6-ec9da60a0902-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:20.137952 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.137856    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37648ec-d503-43af-aef6-ec9da60a0902-kube-api-access-8rh9l" pod "b37648ec-d503-43af-aef6-ec9da60a0902" (UID: "b37648ec-d503-43af-aef6-ec9da60a0902"). InnerVolumeSpecName "kube-api-access-8rh9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:20.137952 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.137912    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/341cd65b-6e79-41b5-996c-1b866e57780c-kube-api-access-b6kcb" pod "341cd65b-6e79-41b5-996c-1b866e57780c" (UID: "341cd65b-6e79-41b5-996c-1b866e57780c"). InnerVolumeSpecName "kube-api-access-b6kcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:20.199849 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.199801    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:20.237749 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.237643    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rh9l\" (UniqueName: \"kubernetes.io/projected/b37648ec-d503-43af-aef6-ec9da60a0902-kube-api-access-8rh9l\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:20.237749 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.237716    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6kcb\" (UniqueName: \"kubernetes.io/projected/341cd65b-6e79-41b5-996c-1b866e57780c-kube-api-access-b6kcb\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:20.272212 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.272177    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"]
Oct 30 23:04:20.272424 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.272401    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh" podUID="df4966ae-49e6-461e-b677-6111da780e93" containerName="istio-proxy" containerID="cri-o://7ddc1f3e950b2fdc2698a2862187414e6680d315a0b97a0f70da00a745880367" gracePeriod=30
Oct 30 23:04:20.430105 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.430061    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:20.639353 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.639323    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 23:04:20.707553 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.707517    4889 generic.go:356] "Generic (PLEG): container finished" podID="24e8fc0c-97b0-47f1-ba4b-5ec430718d31" containerID="533f76e65b54feab7f5111481ed774d33442f3ac754e6eddc48347cda2f0e85b" exitCode=0
Oct 30 23:04:20.707741 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.707576    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"
Oct 30 23:04:20.707741 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.707598    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x" event={"ID":"24e8fc0c-97b0-47f1-ba4b-5ec430718d31","Type":"ContainerDied","Data":"533f76e65b54feab7f5111481ed774d33442f3ac754e6eddc48347cda2f0e85b"}
Oct 30 23:04:20.707741 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.707661    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x" event={"ID":"24e8fc0c-97b0-47f1-ba4b-5ec430718d31","Type":"ContainerDied","Data":"4241fe15034384047bed8db225edc71620ba8960f2b9d463554d7ab72ac02acc"}
Oct 30 23:04:20.707741 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.707689    4889 scope.go:122] "RemoveContainer" containerID="533f76e65b54feab7f5111481ed774d33442f3ac754e6eddc48347cda2f0e85b"
Oct 30 23:04:20.709698 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.709673    4889 generic.go:356] "Generic (PLEG): container finished" podID="341cd65b-6e79-41b5-996c-1b866e57780c" containerID="19a2d32ad49e78a0353850ac47e9854be819d22c5cf249b26249a4cdef9eeeaa" exitCode=0
Oct 30 23:04:20.709818 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.709730    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd"
Oct 30 23:04:20.709886 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.709833    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd" event={"ID":"341cd65b-6e79-41b5-996c-1b866e57780c","Type":"ContainerDied","Data":"19a2d32ad49e78a0353850ac47e9854be819d22c5cf249b26249a4cdef9eeeaa"}
Oct 30 23:04:20.709886 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.709874    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd" event={"ID":"341cd65b-6e79-41b5-996c-1b866e57780c","Type":"ContainerDied","Data":"a800fe6a8039be3f172b17739d13e27ef824ebb2b005f5f0ad2eef0499d2815e"}
Oct 30 23:04:20.723673 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.723500    4889 generic.go:356] "Generic (PLEG): container finished" podID="b37648ec-d503-43af-aef6-ec9da60a0902" containerID="0b0e660e33109897d30c5f9f6d29a88f0f7a8b700186279151228cccb1533f3d" exitCode=0
Oct 30 23:04:20.723673 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.723615    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9"
Oct 30 23:04:20.723673 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.723646    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9" event={"ID":"b37648ec-d503-43af-aef6-ec9da60a0902","Type":"ContainerDied","Data":"0b0e660e33109897d30c5f9f6d29a88f0f7a8b700186279151228cccb1533f3d"}
Oct 30 23:04:20.723864 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.723711    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9" event={"ID":"b37648ec-d503-43af-aef6-ec9da60a0902","Type":"ContainerDied","Data":"976e42b879a10b11250293188940f7c38198305512f92d27b32c2a7756fc0371"}
Oct 30 23:04:20.739944 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.739924    4889 scope.go:122] "RemoveContainer" containerID="533f76e65b54feab7f5111481ed774d33442f3ac754e6eddc48347cda2f0e85b"
Oct 30 23:04:20.740461 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:04:20.740423    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"533f76e65b54feab7f5111481ed774d33442f3ac754e6eddc48347cda2f0e85b\": container with ID starting with 533f76e65b54feab7f5111481ed774d33442f3ac754e6eddc48347cda2f0e85b not found: ID does not exist" containerID="533f76e65b54feab7f5111481ed774d33442f3ac754e6eddc48347cda2f0e85b"
Oct 30 23:04:20.740554 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.740464    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"533f76e65b54feab7f5111481ed774d33442f3ac754e6eddc48347cda2f0e85b"} err="failed to get container status \"533f76e65b54feab7f5111481ed774d33442f3ac754e6eddc48347cda2f0e85b\": rpc error: code = NotFound desc = could not find container \"533f76e65b54feab7f5111481ed774d33442f3ac754e6eddc48347cda2f0e85b\": container with ID starting with 533f76e65b54feab7f5111481ed774d33442f3ac754e6eddc48347cda2f0e85b not found: ID does not exist"
Oct 30 23:04:20.740554 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.740492    4889 scope.go:122] "RemoveContainer" containerID="19a2d32ad49e78a0353850ac47e9854be819d22c5cf249b26249a4cdef9eeeaa"
Oct 30 23:04:20.743915 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.743882    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istiod-ca-cert\") pod \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") "
Oct 30 23:04:20.744030 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.743920    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-token\" (UniqueName: \"kubernetes.io/projected/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-token\") pod \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") "
Oct 30 23:04:20.744030 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.744013    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-workload-certs\") pod \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") "
Oct 30 23:04:20.744395 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.744354    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istiod-ca-cert" pod "24e8fc0c-97b0-47f1-ba4b-5ec430718d31" (UID: "24e8fc0c-97b0-47f1-ba4b-5ec430718d31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:20.744395 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.744344    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-workload-certs" pod "24e8fc0c-97b0-47f1-ba4b-5ec430718d31" (UID: "24e8fc0c-97b0-47f1-ba4b-5ec430718d31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:20.744555 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.744405    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-podinfo\") pod \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") "
Oct 30 23:04:20.744555 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.744483    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-workload-socket\") pod \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") "
Oct 30 23:04:20.744555 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.744525    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-envoy\") pod \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") "
Oct 30 23:04:20.744736 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.744613    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-kube-api-access-mf9xf\" (UniqueName: \"kubernetes.io/projected/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-kube-api-access-mf9xf\") pod \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") "
Oct 30 23:04:20.744736 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.744675    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-credential-socket\") pod \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") "
Oct 30 23:04:20.744736 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.744712    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-data\") pod \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\" (UID: \"24e8fc0c-97b0-47f1-ba4b-5ec430718d31\") "
Oct 30 23:04:20.745240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.745197    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:20.745329 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.745268    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:20.745400 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.745324    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-credential-socket" pod "24e8fc0c-97b0-47f1-ba4b-5ec430718d31" (UID: "24e8fc0c-97b0-47f1-ba4b-5ec430718d31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:20.745479 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.745457    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-data" pod "24e8fc0c-97b0-47f1-ba4b-5ec430718d31" (UID: "24e8fc0c-97b0-47f1-ba4b-5ec430718d31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:20.746881 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.746342    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-workload-socket" pod "24e8fc0c-97b0-47f1-ba4b-5ec430718d31" (UID: "24e8fc0c-97b0-47f1-ba4b-5ec430718d31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:20.746881 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.746670    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-token" pod "24e8fc0c-97b0-47f1-ba4b-5ec430718d31" (UID: "24e8fc0c-97b0-47f1-ba4b-5ec430718d31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:20.747185 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.747162    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-envoy" pod "24e8fc0c-97b0-47f1-ba4b-5ec430718d31" (UID: "24e8fc0c-97b0-47f1-ba4b-5ec430718d31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:20.747728 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.747707    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-podinfo" pod "24e8fc0c-97b0-47f1-ba4b-5ec430718d31" (UID: "24e8fc0c-97b0-47f1-ba4b-5ec430718d31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:04:20.748530 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.748510    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-kube-api-access-mf9xf" pod "24e8fc0c-97b0-47f1-ba4b-5ec430718d31" (UID: "24e8fc0c-97b0-47f1-ba4b-5ec430718d31"). InnerVolumeSpecName "kube-api-access-mf9xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:20.771677 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.771644    4889 scope.go:122] "RemoveContainer" containerID="19a2d32ad49e78a0353850ac47e9854be819d22c5cf249b26249a4cdef9eeeaa"
Oct 30 23:04:20.771965 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:04:20.771944    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19a2d32ad49e78a0353850ac47e9854be819d22c5cf249b26249a4cdef9eeeaa\": container with ID starting with 19a2d32ad49e78a0353850ac47e9854be819d22c5cf249b26249a4cdef9eeeaa not found: ID does not exist" containerID="19a2d32ad49e78a0353850ac47e9854be819d22c5cf249b26249a4cdef9eeeaa"
Oct 30 23:04:20.772047 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.771974    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19a2d32ad49e78a0353850ac47e9854be819d22c5cf249b26249a4cdef9eeeaa"} err="failed to get container status \"19a2d32ad49e78a0353850ac47e9854be819d22c5cf249b26249a4cdef9eeeaa\": rpc error: code = NotFound desc = could not find container \"19a2d32ad49e78a0353850ac47e9854be819d22c5cf249b26249a4cdef9eeeaa\": container with ID starting with 19a2d32ad49e78a0353850ac47e9854be819d22c5cf249b26249a4cdef9eeeaa not found: ID does not exist"
Oct 30 23:04:20.772047 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.771990    4889 scope.go:122] "RemoveContainer" containerID="0b0e660e33109897d30c5f9f6d29a88f0f7a8b700186279151228cccb1533f3d"
Oct 30 23:04:20.782456 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.782398    4889 scope.go:122] "RemoveContainer" containerID="0b0e660e33109897d30c5f9f6d29a88f0f7a8b700186279151228cccb1533f3d"
Oct 30 23:04:20.782819 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:04:20.782797    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b0e660e33109897d30c5f9f6d29a88f0f7a8b700186279151228cccb1533f3d\": container with ID starting with 0b0e660e33109897d30c5f9f6d29a88f0f7a8b700186279151228cccb1533f3d not found: ID does not exist" containerID="0b0e660e33109897d30c5f9f6d29a88f0f7a8b700186279151228cccb1533f3d"
Oct 30 23:04:20.782905 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.782830    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b0e660e33109897d30c5f9f6d29a88f0f7a8b700186279151228cccb1533f3d"} err="failed to get container status \"0b0e660e33109897d30c5f9f6d29a88f0f7a8b700186279151228cccb1533f3d\": rpc error: code = NotFound desc = could not find container \"0b0e660e33109897d30c5f9f6d29a88f0f7a8b700186279151228cccb1533f3d\": container with ID starting with 0b0e660e33109897d30c5f9f6d29a88f0f7a8b700186279151228cccb1533f3d not found: ID does not exist"
Oct 30 23:04:20.796567 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.796529    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9"]
Oct 30 23:04:20.800184 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.800144    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3cd77f-799d6dfd99-2zkl9"]
Oct 30 23:04:20.835494 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.835458    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd"]
Oct 30 23:04:20.840950 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.840915    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3cd77f-6d7cccccf5-gxpwd"]
Oct 30 23:04:20.846155 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.846127    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:20.846155 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.846155    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:20.846364 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.846165    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:20.846364 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.846175    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:20.846364 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.846184    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf9xf\" (UniqueName: \"kubernetes.io/projected/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-kube-api-access-mf9xf\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:20.846364 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.846194    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:20.846364 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:20.846203    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24e8fc0c-97b0-47f1-ba4b-5ec430718d31-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:21.054508 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.054418    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"]
Oct 30 23:04:21.063268 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.063207    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-wsd2x"]
Oct 30 23:04:21.146495 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.146454    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:21.340833 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.340806    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 23:04:21.452770 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.452730    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65b844a4-133d-40a6-bc49-81bbffc94221-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65b844a4-133d-40a6-bc49-81bbffc94221-istiod-ca-cert\") pod \"65b844a4-133d-40a6-bc49-81bbffc94221\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") "
Oct 30 23:04:21.452961 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.452816    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-istio-envoy\") pod \"65b844a4-133d-40a6-bc49-81bbffc94221\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") "
Oct 30 23:04:21.452961 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.452908    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-workload-certs\") pod \"65b844a4-133d-40a6-bc49-81bbffc94221\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") "
Oct 30 23:04:21.452961 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.452937    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-istio-data\") pod \"65b844a4-133d-40a6-bc49-81bbffc94221\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") "
Oct 30 23:04:21.452961 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.452959    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65b844a4-133d-40a6-bc49-81bbffc94221-istio-token\" (UniqueName: \"kubernetes.io/projected/65b844a4-133d-40a6-bc49-81bbffc94221-istio-token\") pod \"65b844a4-133d-40a6-bc49-81bbffc94221\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") "
Oct 30 23:04:21.453171 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.452997    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65b844a4-133d-40a6-bc49-81bbffc94221-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65b844a4-133d-40a6-bc49-81bbffc94221-istio-podinfo\") pod \"65b844a4-133d-40a6-bc49-81bbffc94221\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") "
Oct 30 23:04:21.453171 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.453067    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-workload-socket\") pod \"65b844a4-133d-40a6-bc49-81bbffc94221\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") "
Oct 30 23:04:21.453171 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.453125    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-credential-socket\") pod \"65b844a4-133d-40a6-bc49-81bbffc94221\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") "
Oct 30 23:04:21.453171 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.453156    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65b844a4-133d-40a6-bc49-81bbffc94221-kube-api-access-nfdlq\" (UniqueName: \"kubernetes.io/projected/65b844a4-133d-40a6-bc49-81bbffc94221-kube-api-access-nfdlq\") pod \"65b844a4-133d-40a6-bc49-81bbffc94221\" (UID: \"65b844a4-133d-40a6-bc49-81bbffc94221\") "
Oct 30 23:04:21.453441 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.453202    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65b844a4-133d-40a6-bc49-81bbffc94221-istiod-ca-cert" pod "65b844a4-133d-40a6-bc49-81bbffc94221" (UID: "65b844a4-133d-40a6-bc49-81bbffc94221"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:21.453441 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.453366    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-workload-certs" pod "65b844a4-133d-40a6-bc49-81bbffc94221" (UID: "65b844a4-133d-40a6-bc49-81bbffc94221"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:21.453441 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.453350    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-istio-data" pod "65b844a4-133d-40a6-bc49-81bbffc94221" (UID: "65b844a4-133d-40a6-bc49-81bbffc94221"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:21.453619 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.453513    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-workload-socket" pod "65b844a4-133d-40a6-bc49-81bbffc94221" (UID: "65b844a4-133d-40a6-bc49-81bbffc94221"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:21.453619 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.453582    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-credential-socket" pod "65b844a4-133d-40a6-bc49-81bbffc94221" (UID: "65b844a4-133d-40a6-bc49-81bbffc94221"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:21.453718 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.453684    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:21.453718 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.453707    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:21.453821 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.453723    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65b844a4-133d-40a6-bc49-81bbffc94221-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:21.453821 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.453737    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:21.453821 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.453750    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:21.455467 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.455441    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-istio-envoy" pod "65b844a4-133d-40a6-bc49-81bbffc94221" (UID: "65b844a4-133d-40a6-bc49-81bbffc94221"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:21.455608 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.455444    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65b844a4-133d-40a6-bc49-81bbffc94221-istio-podinfo" pod "65b844a4-133d-40a6-bc49-81bbffc94221" (UID: "65b844a4-133d-40a6-bc49-81bbffc94221"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:04:21.455608 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.455529    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b844a4-133d-40a6-bc49-81bbffc94221-kube-api-access-nfdlq" pod "65b844a4-133d-40a6-bc49-81bbffc94221" (UID: "65b844a4-133d-40a6-bc49-81bbffc94221"). InnerVolumeSpecName "kube-api-access-nfdlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:21.455845 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.455826    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b844a4-133d-40a6-bc49-81bbffc94221-istio-token" pod "65b844a4-133d-40a6-bc49-81bbffc94221" (UID: "65b844a4-133d-40a6-bc49-81bbffc94221"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:21.511863 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.511826    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"]
Oct 30 23:04:21.554547 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.554513    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfdlq\" (UniqueName: \"kubernetes.io/projected/65b844a4-133d-40a6-bc49-81bbffc94221-kube-api-access-nfdlq\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:21.554547 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.554542    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65b844a4-133d-40a6-bc49-81bbffc94221-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:21.554547 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.554555    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65b844a4-133d-40a6-bc49-81bbffc94221-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:21.554794 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.554563    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65b844a4-133d-40a6-bc49-81bbffc94221-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:21.717615 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.717501    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:21.731442 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.731409    4889 generic.go:356] "Generic (PLEG): container finished" podID="65b844a4-133d-40a6-bc49-81bbffc94221" containerID="6cbde591fc250ce6ef68b813f77a7c02533d36d7bb56a89dd70ce9992f5eedc8" exitCode=0
Oct 30 23:04:21.731597 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.731459    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"
Oct 30 23:04:21.731597 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.731470    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp" event={"ID":"65b844a4-133d-40a6-bc49-81bbffc94221","Type":"ContainerDied","Data":"6cbde591fc250ce6ef68b813f77a7c02533d36d7bb56a89dd70ce9992f5eedc8"}
Oct 30 23:04:21.731597 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.731531    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp" event={"ID":"65b844a4-133d-40a6-bc49-81bbffc94221","Type":"ContainerDied","Data":"d7c8f696bed537c96e3fefee649fc8b5444e2edd83c2cc3ab83ddb957ac777d1"}
Oct 30 23:04:21.731597 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.731560    4889 scope.go:122] "RemoveContainer" containerID="6cbde591fc250ce6ef68b813f77a7c02533d36d7bb56a89dd70ce9992f5eedc8"
Oct 30 23:04:21.742941 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.742919    4889 scope.go:122] "RemoveContainer" containerID="6cbde591fc250ce6ef68b813f77a7c02533d36d7bb56a89dd70ce9992f5eedc8"
Oct 30 23:04:21.743215 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:04:21.743194    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cbde591fc250ce6ef68b813f77a7c02533d36d7bb56a89dd70ce9992f5eedc8\": container with ID starting with 6cbde591fc250ce6ef68b813f77a7c02533d36d7bb56a89dd70ce9992f5eedc8 not found: ID does not exist" containerID="6cbde591fc250ce6ef68b813f77a7c02533d36d7bb56a89dd70ce9992f5eedc8"
Oct 30 23:04:21.743302 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.743239    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cbde591fc250ce6ef68b813f77a7c02533d36d7bb56a89dd70ce9992f5eedc8"} err="failed to get container status \"6cbde591fc250ce6ef68b813f77a7c02533d36d7bb56a89dd70ce9992f5eedc8\": rpc error: code = NotFound desc = could not find container \"6cbde591fc250ce6ef68b813f77a7c02533d36d7bb56a89dd70ce9992f5eedc8\": container with ID starting with 6cbde591fc250ce6ef68b813f77a7c02533d36d7bb56a89dd70ce9992f5eedc8 not found: ID does not exist"
Oct 30 23:04:21.787960 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.787921    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"]
Oct 30 23:04:21.797426 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:21.797396    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-p2chp"]
Oct 30 23:04:22.136582 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:22.136547    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:22.324754 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:22.324716    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:22.779678 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:22.779632    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24e8fc0c-97b0-47f1-ba4b-5ec430718d31" path="/var/lib/kubelet/pods/24e8fc0c-97b0-47f1-ba4b-5ec430718d31/volumes"
Oct 30 23:04:22.780278 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:22.780255    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="341cd65b-6e79-41b5-996c-1b866e57780c" path="/var/lib/kubelet/pods/341cd65b-6e79-41b5-996c-1b866e57780c/volumes"
Oct 30 23:04:22.780743 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:22.780724    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65b844a4-133d-40a6-bc49-81bbffc94221" path="/var/lib/kubelet/pods/65b844a4-133d-40a6-bc49-81bbffc94221/volumes"
Oct 30 23:04:22.781577 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:22.781554    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b37648ec-d503-43af-aef6-ec9da60a0902" path="/var/lib/kubelet/pods/b37648ec-d503-43af-aef6-ec9da60a0902/volumes"
Oct 30 23:04:23.608857 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:23.608819    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:23.790209 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:23.790171    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:23.967741 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:23.967647    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:24.440863 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:24.440815    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:25.152400 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.152349    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:25.336594 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.336560    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:25.400968 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.396734    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"]
Oct 30 23:04:25.400968 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.397090    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj" podUID="d7ce41e7-a1cb-4f49-a247-da6c57d34d53" containerName="istio-proxy" containerID="cri-o://f58f796232e084b576c3ee3532918fcd6300af0b24873b92657698eb0826aac5" gracePeriod=30
Oct 30 23:04:25.512041 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.512000    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:25.566033 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.566002    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 23:04:25.678517 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.678476    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:25.698614 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.698584    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df4966ae-49e6-461e-b677-6111da780e93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4966ae-49e6-461e-b677-6111da780e93-istio-podinfo\") pod \"df4966ae-49e6-461e-b677-6111da780e93\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") "
Oct 30 23:04:25.698819 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.698660    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df4966ae-49e6-461e-b677-6111da780e93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4966ae-49e6-461e-b677-6111da780e93-istiod-ca-cert\") pod \"df4966ae-49e6-461e-b677-6111da780e93\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") "
Oct 30 23:04:25.698819 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.698785    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df4966ae-49e6-461e-b677-6111da780e93-kube-api-access-m6s6s\" (UniqueName: \"kubernetes.io/projected/df4966ae-49e6-461e-b677-6111da780e93-kube-api-access-m6s6s\") pod \"df4966ae-49e6-461e-b677-6111da780e93\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") "
Oct 30 23:04:25.698819 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.698816    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-workload-certs\") pod \"df4966ae-49e6-461e-b677-6111da780e93\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") "
Oct 30 23:04:25.699002 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.698854    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-istio-envoy\") pod \"df4966ae-49e6-461e-b677-6111da780e93\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") "
Oct 30 23:04:25.699002 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.698902    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-workload-socket\") pod \"df4966ae-49e6-461e-b677-6111da780e93\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") "
Oct 30 23:04:25.699002 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.698938    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df4966ae-49e6-461e-b677-6111da780e93-istio-token\" (UniqueName: \"kubernetes.io/projected/df4966ae-49e6-461e-b677-6111da780e93-istio-token\") pod \"df4966ae-49e6-461e-b677-6111da780e93\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") "
Oct 30 23:04:25.699002 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.698971    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-credential-socket\") pod \"df4966ae-49e6-461e-b677-6111da780e93\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") "
Oct 30 23:04:25.699217 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.699049    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-istio-data\") pod \"df4966ae-49e6-461e-b677-6111da780e93\" (UID: \"df4966ae-49e6-461e-b677-6111da780e93\") "
Oct 30 23:04:25.699217 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.699072    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-workload-certs" pod "df4966ae-49e6-461e-b677-6111da780e93" (UID: "df4966ae-49e6-461e-b677-6111da780e93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:25.699217 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.699109    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df4966ae-49e6-461e-b677-6111da780e93-istiod-ca-cert" pod "df4966ae-49e6-461e-b677-6111da780e93" (UID: "df4966ae-49e6-461e-b677-6111da780e93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:25.699443 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.699295    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-workload-socket" pod "df4966ae-49e6-461e-b677-6111da780e93" (UID: "df4966ae-49e6-461e-b677-6111da780e93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:25.699637 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.699559    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-istio-data" pod "df4966ae-49e6-461e-b677-6111da780e93" (UID: "df4966ae-49e6-461e-b677-6111da780e93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:25.699899 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.699647    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4966ae-49e6-461e-b677-6111da780e93-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:25.699899 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.699670    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:25.699899 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.699684    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:25.700098 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.699780    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-credential-socket" pod "df4966ae-49e6-461e-b677-6111da780e93" (UID: "df4966ae-49e6-461e-b677-6111da780e93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:25.701545 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.701520    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-istio-envoy" pod "df4966ae-49e6-461e-b677-6111da780e93" (UID: "df4966ae-49e6-461e-b677-6111da780e93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:25.701545 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.701546    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df4966ae-49e6-461e-b677-6111da780e93-istio-podinfo" pod "df4966ae-49e6-461e-b677-6111da780e93" (UID: "df4966ae-49e6-461e-b677-6111da780e93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:04:25.701545 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.701558    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df4966ae-49e6-461e-b677-6111da780e93-istio-token" pod "df4966ae-49e6-461e-b677-6111da780e93" (UID: "df4966ae-49e6-461e-b677-6111da780e93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:25.701822 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.701725    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df4966ae-49e6-461e-b677-6111da780e93-kube-api-access-m6s6s" pod "df4966ae-49e6-461e-b677-6111da780e93" (UID: "df4966ae-49e6-461e-b677-6111da780e93"). InnerVolumeSpecName "kube-api-access-m6s6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:25.761552 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.761516    4889 generic.go:356] "Generic (PLEG): container finished" podID="df4966ae-49e6-461e-b677-6111da780e93" containerID="7ddc1f3e950b2fdc2698a2862187414e6680d315a0b97a0f70da00a745880367" exitCode=0
Oct 30 23:04:25.761763 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.761564    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"
Oct 30 23:04:25.761763 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.761609    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh" event={"ID":"df4966ae-49e6-461e-b677-6111da780e93","Type":"ContainerDied","Data":"7ddc1f3e950b2fdc2698a2862187414e6680d315a0b97a0f70da00a745880367"}
Oct 30 23:04:25.761763 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.761683    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh" event={"ID":"df4966ae-49e6-461e-b677-6111da780e93","Type":"ContainerDied","Data":"d3ca3be2017c46576d1c251a61638a4fcebb3e5eb5d87f44e21a7e3e814940ee"}
Oct 30 23:04:25.761763 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.761712    4889 scope.go:122] "RemoveContainer" containerID="7ddc1f3e950b2fdc2698a2862187414e6680d315a0b97a0f70da00a745880367"
Oct 30 23:04:25.773314 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.773292    4889 scope.go:122] "RemoveContainer" containerID="2f9d5c5f3c357c6ed6f1e5fe72ee40e13646424e2f2b8dd53d6cb0e392d8d665"
Oct 30 23:04:25.782499 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.782479    4889 scope.go:122] "RemoveContainer" containerID="7ddc1f3e950b2fdc2698a2862187414e6680d315a0b97a0f70da00a745880367"
Oct 30 23:04:25.782782 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:04:25.782762    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ddc1f3e950b2fdc2698a2862187414e6680d315a0b97a0f70da00a745880367\": container with ID starting with 7ddc1f3e950b2fdc2698a2862187414e6680d315a0b97a0f70da00a745880367 not found: ID does not exist" containerID="7ddc1f3e950b2fdc2698a2862187414e6680d315a0b97a0f70da00a745880367"
Oct 30 23:04:25.782832 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.782792    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ddc1f3e950b2fdc2698a2862187414e6680d315a0b97a0f70da00a745880367"} err="failed to get container status \"7ddc1f3e950b2fdc2698a2862187414e6680d315a0b97a0f70da00a745880367\": rpc error: code = NotFound desc = could not find container \"7ddc1f3e950b2fdc2698a2862187414e6680d315a0b97a0f70da00a745880367\": container with ID starting with 7ddc1f3e950b2fdc2698a2862187414e6680d315a0b97a0f70da00a745880367 not found: ID does not exist"
Oct 30 23:04:25.782832 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.782810    4889 scope.go:122] "RemoveContainer" containerID="2f9d5c5f3c357c6ed6f1e5fe72ee40e13646424e2f2b8dd53d6cb0e392d8d665"
Oct 30 23:04:25.783040 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:04:25.783020    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f9d5c5f3c357c6ed6f1e5fe72ee40e13646424e2f2b8dd53d6cb0e392d8d665\": container with ID starting with 2f9d5c5f3c357c6ed6f1e5fe72ee40e13646424e2f2b8dd53d6cb0e392d8d665 not found: ID does not exist" containerID="2f9d5c5f3c357c6ed6f1e5fe72ee40e13646424e2f2b8dd53d6cb0e392d8d665"
Oct 30 23:04:25.783085 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.783046    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f9d5c5f3c357c6ed6f1e5fe72ee40e13646424e2f2b8dd53d6cb0e392d8d665"} err="failed to get container status \"2f9d5c5f3c357c6ed6f1e5fe72ee40e13646424e2f2b8dd53d6cb0e392d8d665\": rpc error: code = NotFound desc = could not find container \"2f9d5c5f3c357c6ed6f1e5fe72ee40e13646424e2f2b8dd53d6cb0e392d8d665\": container with ID starting with 2f9d5c5f3c357c6ed6f1e5fe72ee40e13646424e2f2b8dd53d6cb0e392d8d665 not found: ID does not exist"
Oct 30 23:04:25.801271 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.801236    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:25.801271 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.801263    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4966ae-49e6-461e-b677-6111da780e93-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:25.801271 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.801272    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6s6s\" (UniqueName: \"kubernetes.io/projected/df4966ae-49e6-461e-b677-6111da780e93-kube-api-access-m6s6s\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:25.801491 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.801282    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:25.801491 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.801290    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df4966ae-49e6-461e-b677-6111da780e93-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:25.801491 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.801298    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4966ae-49e6-461e-b677-6111da780e93-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:25.817060 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.817028    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"]
Oct 30 23:04:25.821369 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.821338    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f94561-openshift-default-66c966685c-zcxmh"]
Oct 30 23:04:25.966256 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:25.966153    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:26.773824 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:26.773778    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df4966ae-49e6-461e-b677-6111da780e93" path="/var/lib/kubelet/pods/df4966ae-49e6-461e-b677-6111da780e93/volumes"
Oct 30 23:04:27.251889 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.251855    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7"]
Oct 30 23:04:27.252130 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.252106    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7" podUID="ddfac372-f0f0-4e7b-a661-007adc297ebd" containerName="maas-api" containerID="cri-o://f5811c9e77f137eae0830817988c7e9c3107c3c8832b287e9112cecc96c2f157" gracePeriod=30
Oct 30 23:04:27.379569 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.379513    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:27.506187 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.506126    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7"
Oct 30 23:04:27.586183 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.586148    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:27.621195 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.621150    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddfac372-f0f0-4e7b-a661-007adc297ebd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddfac372-f0f0-4e7b-a661-007adc297ebd-metrics-certs\") pod \"ddfac372-f0f0-4e7b-a661-007adc297ebd\" (UID: \"ddfac372-f0f0-4e7b-a661-007adc297ebd\") "
Oct 30 23:04:27.621396 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.621208    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddfac372-f0f0-4e7b-a661-007adc297ebd-kube-api-access-sxqq9\" (UniqueName: \"kubernetes.io/projected/ddfac372-f0f0-4e7b-a661-007adc297ebd-kube-api-access-sxqq9\") pod \"ddfac372-f0f0-4e7b-a661-007adc297ebd\" (UID: \"ddfac372-f0f0-4e7b-a661-007adc297ebd\") "
Oct 30 23:04:27.621396 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.621309    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddfac372-f0f0-4e7b-a661-007adc297ebd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddfac372-f0f0-4e7b-a661-007adc297ebd-openshift-service-ca\") pod \"ddfac372-f0f0-4e7b-a661-007adc297ebd\" (UID: \"ddfac372-f0f0-4e7b-a661-007adc297ebd\") "
Oct 30 23:04:27.621509 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.621397    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddfac372-f0f0-4e7b-a661-007adc297ebd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddfac372-f0f0-4e7b-a661-007adc297ebd-maas-api-tls\") pod \"ddfac372-f0f0-4e7b-a661-007adc297ebd\" (UID: \"ddfac372-f0f0-4e7b-a661-007adc297ebd\") "
Oct 30 23:04:27.621702 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.621670    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddfac372-f0f0-4e7b-a661-007adc297ebd-openshift-service-ca" pod "ddfac372-f0f0-4e7b-a661-007adc297ebd" (UID: "ddfac372-f0f0-4e7b-a661-007adc297ebd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:27.621948 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.621921    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddfac372-f0f0-4e7b-a661-007adc297ebd-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:27.623636 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.623612    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddfac372-f0f0-4e7b-a661-007adc297ebd-kube-api-access-sxqq9" pod "ddfac372-f0f0-4e7b-a661-007adc297ebd" (UID: "ddfac372-f0f0-4e7b-a661-007adc297ebd"). InnerVolumeSpecName "kube-api-access-sxqq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:27.624209 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.624188    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddfac372-f0f0-4e7b-a661-007adc297ebd-maas-api-tls" pod "ddfac372-f0f0-4e7b-a661-007adc297ebd" (UID: "ddfac372-f0f0-4e7b-a661-007adc297ebd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:27.624307 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.624214    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddfac372-f0f0-4e7b-a661-007adc297ebd-metrics-certs" pod "ddfac372-f0f0-4e7b-a661-007adc297ebd" (UID: "ddfac372-f0f0-4e7b-a661-007adc297ebd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:27.723495 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.723464    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddfac372-f0f0-4e7b-a661-007adc297ebd-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:27.723495 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.723492    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddfac372-f0f0-4e7b-a661-007adc297ebd-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:27.723495 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.723502    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxqq9\" (UniqueName: \"kubernetes.io/projected/ddfac372-f0f0-4e7b-a661-007adc297ebd-kube-api-access-sxqq9\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:27.777263 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.777125    4889 generic.go:356] "Generic (PLEG): container finished" podID="ddfac372-f0f0-4e7b-a661-007adc297ebd" containerID="f5811c9e77f137eae0830817988c7e9c3107c3c8832b287e9112cecc96c2f157" exitCode=0
Oct 30 23:04:27.777263 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.777256    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7"
Oct 30 23:04:27.777825 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.777266    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7" event={"ID":"ddfac372-f0f0-4e7b-a661-007adc297ebd","Type":"ContainerDied","Data":"f5811c9e77f137eae0830817988c7e9c3107c3c8832b287e9112cecc96c2f157"}
Oct 30 23:04:27.777825 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.777340    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7" event={"ID":"ddfac372-f0f0-4e7b-a661-007adc297ebd","Type":"ContainerDied","Data":"1c1e1112f4397bec5f95c4db1dd987e682fc97e45466fd5726dc229a507c16e1"}
Oct 30 23:04:27.777825 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.777366    4889 scope.go:122] "RemoveContainer" containerID="f5811c9e77f137eae0830817988c7e9c3107c3c8832b287e9112cecc96c2f157"
Oct 30 23:04:27.791026 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.790996    4889 scope.go:122] "RemoveContainer" containerID="f5811c9e77f137eae0830817988c7e9c3107c3c8832b287e9112cecc96c2f157"
Oct 30 23:04:27.792039 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:04:27.792014    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5811c9e77f137eae0830817988c7e9c3107c3c8832b287e9112cecc96c2f157\": container with ID starting with f5811c9e77f137eae0830817988c7e9c3107c3c8832b287e9112cecc96c2f157 not found: ID does not exist" containerID="f5811c9e77f137eae0830817988c7e9c3107c3c8832b287e9112cecc96c2f157"
Oct 30 23:04:27.792133 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.792047    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5811c9e77f137eae0830817988c7e9c3107c3c8832b287e9112cecc96c2f157"} err="failed to get container status \"f5811c9e77f137eae0830817988c7e9c3107c3c8832b287e9112cecc96c2f157\": rpc error: code = NotFound desc = could not find container \"f5811c9e77f137eae0830817988c7e9c3107c3c8832b287e9112cecc96c2f157\": container with ID starting with f5811c9e77f137eae0830817988c7e9c3107c3c8832b287e9112cecc96c2f157 not found: ID does not exist"
Oct 30 23:04:27.801843 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.801822    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:27.839849 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.839815    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7"]
Oct 30 23:04:27.853590 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.853561    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-54114e-5648d4c476-vsjq7"]
Oct 30 23:04:27.987127 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:27.987082    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:28.160036 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:28.159931    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:28.339830 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:28.339799    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:28.666949 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:28.666910    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:28.773945 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:28.773910    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddfac372-f0f0-4e7b-a661-007adc297ebd" path="/var/lib/kubelet/pods/ddfac372-f0f0-4e7b-a661-007adc297ebd/volumes"
Oct 30 23:04:29.329873 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:29.329835    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"]
Oct 30 23:04:29.344550 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:29.344512    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"]
Oct 30 23:04:29.506792 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:29.506747    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:30.323883 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.323845    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:30.657855 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.657831    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 23:04:30.751274 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.751217    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-data\") pod \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") "
Oct 30 23:04:30.751490 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.751315    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-token\" (UniqueName: \"kubernetes.io/projected/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-token\") pod \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") "
Oct 30 23:04:30.751490 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.751350    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istiod-ca-cert\") pod \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") "
Oct 30 23:04:30.751490 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.751388    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-envoy\") pod \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") "
Oct 30 23:04:30.751490 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.751426    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-workload-certs\") pod \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") "
Oct 30 23:04:30.751490 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.751452    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-credential-socket\") pod \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") "
Oct 30 23:04:30.751805 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.751556    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-workload-socket\") pod \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") "
Oct 30 23:04:30.751805 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.751629    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-kube-api-access-x5dtk\" (UniqueName: \"kubernetes.io/projected/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-kube-api-access-x5dtk\") pod \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") "
Oct 30 23:04:30.751805 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.751661    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-podinfo\") pod \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\" (UID: \"d7ce41e7-a1cb-4f49-a247-da6c57d34d53\") "
Oct 30 23:04:30.751983 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.751557    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-data" pod "d7ce41e7-a1cb-4f49-a247-da6c57d34d53" (UID: "d7ce41e7-a1cb-4f49-a247-da6c57d34d53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:30.751983 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.751831    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-credential-socket" pod "d7ce41e7-a1cb-4f49-a247-da6c57d34d53" (UID: "d7ce41e7-a1cb-4f49-a247-da6c57d34d53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:30.751983 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.751840    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istiod-ca-cert" pod "d7ce41e7-a1cb-4f49-a247-da6c57d34d53" (UID: "d7ce41e7-a1cb-4f49-a247-da6c57d34d53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:30.752282 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.752255    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:30.752598 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.752288    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:30.752598 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.752303    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:30.752598 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.752259    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-workload-socket" pod "d7ce41e7-a1cb-4f49-a247-da6c57d34d53" (UID: "d7ce41e7-a1cb-4f49-a247-da6c57d34d53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:30.752598 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.752306    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-workload-certs" pod "d7ce41e7-a1cb-4f49-a247-da6c57d34d53" (UID: "d7ce41e7-a1cb-4f49-a247-da6c57d34d53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:30.754218 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.754187    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-podinfo" pod "d7ce41e7-a1cb-4f49-a247-da6c57d34d53" (UID: "d7ce41e7-a1cb-4f49-a247-da6c57d34d53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:04:30.754401 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.754370    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-kube-api-access-x5dtk" pod "d7ce41e7-a1cb-4f49-a247-da6c57d34d53" (UID: "d7ce41e7-a1cb-4f49-a247-da6c57d34d53"). InnerVolumeSpecName "kube-api-access-x5dtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:30.754497 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.754456    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-token" pod "d7ce41e7-a1cb-4f49-a247-da6c57d34d53" (UID: "d7ce41e7-a1cb-4f49-a247-da6c57d34d53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:30.754497 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.754462    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-envoy" pod "d7ce41e7-a1cb-4f49-a247-da6c57d34d53" (UID: "d7ce41e7-a1cb-4f49-a247-da6c57d34d53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:30.797799 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.797762    4889 generic.go:356] "Generic (PLEG): container finished" podID="d7ce41e7-a1cb-4f49-a247-da6c57d34d53" containerID="f58f796232e084b576c3ee3532918fcd6300af0b24873b92657698eb0826aac5" exitCode=0
Oct 30 23:04:30.797971 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.797831    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"
Oct 30 23:04:30.797971 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.797853    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj" event={"ID":"d7ce41e7-a1cb-4f49-a247-da6c57d34d53","Type":"ContainerDied","Data":"f58f796232e084b576c3ee3532918fcd6300af0b24873b92657698eb0826aac5"}
Oct 30 23:04:30.797971 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.797955    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj" event={"ID":"d7ce41e7-a1cb-4f49-a247-da6c57d34d53","Type":"ContainerDied","Data":"32dadc9915f448544eeedcce8766c3806dedfbc3acfa092e9d25f968273fac51"}
Oct 30 23:04:30.798112 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.797985    4889 scope.go:122] "RemoveContainer" containerID="f58f796232e084b576c3ee3532918fcd6300af0b24873b92657698eb0826aac5"
Oct 30 23:04:30.808475 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.808450    4889 scope.go:122] "RemoveContainer" containerID="f52b786cc3ba6ab73873173d3b67ec58ee39c663f597f780bb9e70b04ae8225a"
Oct 30 23:04:30.818164 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.818137    4889 scope.go:122] "RemoveContainer" containerID="f58f796232e084b576c3ee3532918fcd6300af0b24873b92657698eb0826aac5"
Oct 30 23:04:30.818568 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:04:30.818541    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f58f796232e084b576c3ee3532918fcd6300af0b24873b92657698eb0826aac5\": container with ID starting with f58f796232e084b576c3ee3532918fcd6300af0b24873b92657698eb0826aac5 not found: ID does not exist" containerID="f58f796232e084b576c3ee3532918fcd6300af0b24873b92657698eb0826aac5"
Oct 30 23:04:30.818638 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.818583    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f58f796232e084b576c3ee3532918fcd6300af0b24873b92657698eb0826aac5"} err="failed to get container status \"f58f796232e084b576c3ee3532918fcd6300af0b24873b92657698eb0826aac5\": rpc error: code = NotFound desc = could not find container \"f58f796232e084b576c3ee3532918fcd6300af0b24873b92657698eb0826aac5\": container with ID starting with f58f796232e084b576c3ee3532918fcd6300af0b24873b92657698eb0826aac5 not found: ID does not exist"
Oct 30 23:04:30.818638 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.818611    4889 scope.go:122] "RemoveContainer" containerID="f52b786cc3ba6ab73873173d3b67ec58ee39c663f597f780bb9e70b04ae8225a"
Oct 30 23:04:30.818942 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:04:30.818922    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f52b786cc3ba6ab73873173d3b67ec58ee39c663f597f780bb9e70b04ae8225a\": container with ID starting with f52b786cc3ba6ab73873173d3b67ec58ee39c663f597f780bb9e70b04ae8225a not found: ID does not exist" containerID="f52b786cc3ba6ab73873173d3b67ec58ee39c663f597f780bb9e70b04ae8225a"
Oct 30 23:04:30.818995 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.818953    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52b786cc3ba6ab73873173d3b67ec58ee39c663f597f780bb9e70b04ae8225a"} err="failed to get container status \"f52b786cc3ba6ab73873173d3b67ec58ee39c663f597f780bb9e70b04ae8225a\": rpc error: code = NotFound desc = could not find container \"f52b786cc3ba6ab73873173d3b67ec58ee39c663f597f780bb9e70b04ae8225a\": container with ID starting with f52b786cc3ba6ab73873173d3b67ec58ee39c663f597f780bb9e70b04ae8225a not found: ID does not exist"
Oct 30 23:04:30.853688 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.853650    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:30.853688 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.853678    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:30.853688 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.853688    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:30.854014 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.853699    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:30.854014 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.853714    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5dtk\" (UniqueName: \"kubernetes.io/projected/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-kube-api-access-x5dtk\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:30.854014 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.853726    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ce41e7-a1cb-4f49-a247-da6c57d34d53-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:30.854797 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.854773    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"]
Oct 30 23:04:30.860321 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.860284    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-ztnmj"]
Oct 30 23:04:30.864174 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.864136    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"]
Oct 30 23:04:30.878020 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:30.877984    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"]
Oct 30 23:04:31.018437 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:31.018401    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:32.273036 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:32.272986    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:32.774405 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:32.774369    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7ce41e7-a1cb-4f49-a247-da6c57d34d53" path="/var/lib/kubelet/pods/d7ce41e7-a1cb-4f49-a247-da6c57d34d53/volumes"
Oct 30 23:04:33.448917 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:33.448876    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:34.022907 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:34.022863    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:35.251625 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.251547    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:35.416686 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.416641    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:35.472939 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.472907    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt"]
Oct 30 23:04:35.473150 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.473121    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt" podUID="bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048" containerName="payload-processing" containerID="cri-o://75c6e610d15f0fe7228dcd0cc2dc8064165a72af7290f8e476e72229edf71209" gracePeriod=30
Oct 30 23:04:35.506254 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.506167    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt"]
Oct 30 23:04:35.506436 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.506411    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt" podUID="862bf26d-62d5-4a6f-ba47-242712b43d0a" containerName="payload-pre-processing" containerID="cri-o://992dbf6f54a6c6ad1a9714ea906316fa62c46afbaa0fc2fdb67960c666e3d80a" gracePeriod=30
Oct 30 23:04:35.586557 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.586519    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:35.785943 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.785881    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt"
Oct 30 23:04:35.797816 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.797788    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048-plugins-config-volume\") pod \"bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048\" (UID: \"bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048\") "
Oct 30 23:04:35.797995 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.797843    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048-kube-api-access-rm4md\" (UniqueName: \"kubernetes.io/projected/bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048-kube-api-access-rm4md\") pod \"bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048\" (UID: \"bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048\") "
Oct 30 23:04:35.798202 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.798180    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048-plugins-config-volume" pod "bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048" (UID: "bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:35.800532 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.800504    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048-kube-api-access-rm4md" pod "bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048" (UID: "bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048"). InnerVolumeSpecName "kube-api-access-rm4md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:35.821674 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.821647    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt"
Oct 30 23:04:35.838838 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.830126    4889 generic.go:356] "Generic (PLEG): container finished" podID="bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048" containerID="75c6e610d15f0fe7228dcd0cc2dc8064165a72af7290f8e476e72229edf71209" exitCode=0
Oct 30 23:04:35.838838 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.830330    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt" event={"ID":"bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048","Type":"ContainerDied","Data":"75c6e610d15f0fe7228dcd0cc2dc8064165a72af7290f8e476e72229edf71209"}
Oct 30 23:04:35.838838 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.830414    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt" event={"ID":"bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048","Type":"ContainerDied","Data":"ae308c2e16a0dc39e2997433ac138d27c820869282c5eafbb5b26a1c5329da68"}
Oct 30 23:04:35.838838 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.830448    4889 scope.go:122] "RemoveContainer" containerID="75c6e610d15f0fe7228dcd0cc2dc8064165a72af7290f8e476e72229edf71209"
Oct 30 23:04:35.838838 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.830669    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt"
Oct 30 23:04:35.838838 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.835861    4889 generic.go:356] "Generic (PLEG): container finished" podID="862bf26d-62d5-4a6f-ba47-242712b43d0a" containerID="992dbf6f54a6c6ad1a9714ea906316fa62c46afbaa0fc2fdb67960c666e3d80a" exitCode=0
Oct 30 23:04:35.838838 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.836019    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt" event={"ID":"862bf26d-62d5-4a6f-ba47-242712b43d0a","Type":"ContainerDied","Data":"992dbf6f54a6c6ad1a9714ea906316fa62c46afbaa0fc2fdb67960c666e3d80a"}
Oct 30 23:04:35.838838 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.836064    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt" event={"ID":"862bf26d-62d5-4a6f-ba47-242712b43d0a","Type":"ContainerDied","Data":"bf46df764cca4a043cf48e900d4936d3b692ac3b16b86ebbef2b7363f47ba953"}
Oct 30 23:04:35.838838 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.836133    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt"
Oct 30 23:04:35.849127 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.849085    4889 scope.go:122] "RemoveContainer" containerID="75c6e610d15f0fe7228dcd0cc2dc8064165a72af7290f8e476e72229edf71209"
Oct 30 23:04:35.849639 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:04:35.849616    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75c6e610d15f0fe7228dcd0cc2dc8064165a72af7290f8e476e72229edf71209\": container with ID starting with 75c6e610d15f0fe7228dcd0cc2dc8064165a72af7290f8e476e72229edf71209 not found: ID does not exist" containerID="75c6e610d15f0fe7228dcd0cc2dc8064165a72af7290f8e476e72229edf71209"
Oct 30 23:04:35.849743 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.849648    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c6e610d15f0fe7228dcd0cc2dc8064165a72af7290f8e476e72229edf71209"} err="failed to get container status \"75c6e610d15f0fe7228dcd0cc2dc8064165a72af7290f8e476e72229edf71209\": rpc error: code = NotFound desc = could not find container \"75c6e610d15f0fe7228dcd0cc2dc8064165a72af7290f8e476e72229edf71209\": container with ID starting with 75c6e610d15f0fe7228dcd0cc2dc8064165a72af7290f8e476e72229edf71209 not found: ID does not exist"
Oct 30 23:04:35.849743 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.849670    4889 scope.go:122] "RemoveContainer" containerID="992dbf6f54a6c6ad1a9714ea906316fa62c46afbaa0fc2fdb67960c666e3d80a"
Oct 30 23:04:35.853568 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.853542    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:35.860993 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.860970    4889 scope.go:122] "RemoveContainer" containerID="992dbf6f54a6c6ad1a9714ea906316fa62c46afbaa0fc2fdb67960c666e3d80a"
Oct 30 23:04:35.861320 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:04:35.861299    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"992dbf6f54a6c6ad1a9714ea906316fa62c46afbaa0fc2fdb67960c666e3d80a\": container with ID starting with 992dbf6f54a6c6ad1a9714ea906316fa62c46afbaa0fc2fdb67960c666e3d80a not found: ID does not exist" containerID="992dbf6f54a6c6ad1a9714ea906316fa62c46afbaa0fc2fdb67960c666e3d80a"
Oct 30 23:04:35.861402 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.861328    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"992dbf6f54a6c6ad1a9714ea906316fa62c46afbaa0fc2fdb67960c666e3d80a"} err="failed to get container status \"992dbf6f54a6c6ad1a9714ea906316fa62c46afbaa0fc2fdb67960c666e3d80a\": rpc error: code = NotFound desc = could not find container \"992dbf6f54a6c6ad1a9714ea906316fa62c46afbaa0fc2fdb67960c666e3d80a\": container with ID starting with 992dbf6f54a6c6ad1a9714ea906316fa62c46afbaa0fc2fdb67960c666e3d80a not found: ID does not exist"
Oct 30 23:04:35.898658 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.898626    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/862bf26d-62d5-4a6f-ba47-242712b43d0a-kube-api-access-b9nh7\" (UniqueName: \"kubernetes.io/projected/862bf26d-62d5-4a6f-ba47-242712b43d0a-kube-api-access-b9nh7\") pod \"862bf26d-62d5-4a6f-ba47-242712b43d0a\" (UID: \"862bf26d-62d5-4a6f-ba47-242712b43d0a\") "
Oct 30 23:04:35.898838 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.898730    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/862bf26d-62d5-4a6f-ba47-242712b43d0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862bf26d-62d5-4a6f-ba47-242712b43d0a-plugins-config-volume\") pod \"862bf26d-62d5-4a6f-ba47-242712b43d0a\" (UID: \"862bf26d-62d5-4a6f-ba47-242712b43d0a\") "
Oct 30 23:04:35.898990 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.898977    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:35.899043 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.898992    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm4md\" (UniqueName: \"kubernetes.io/projected/bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048-kube-api-access-rm4md\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:35.899084 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.899063    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/862bf26d-62d5-4a6f-ba47-242712b43d0a-plugins-config-volume" pod "862bf26d-62d5-4a6f-ba47-242712b43d0a" (UID: "862bf26d-62d5-4a6f-ba47-242712b43d0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:35.900883 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.900862    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/862bf26d-62d5-4a6f-ba47-242712b43d0a-kube-api-access-b9nh7" pod "862bf26d-62d5-4a6f-ba47-242712b43d0a" (UID: "862bf26d-62d5-4a6f-ba47-242712b43d0a"). InnerVolumeSpecName "kube-api-access-b9nh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:35.924094 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.924063    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt"]
Oct 30 23:04:35.937998 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:35.932810    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bff950-958c95556-hrzxt"]
Oct 30 23:04:36.000121 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.000079    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9nh7\" (UniqueName: \"kubernetes.io/projected/862bf26d-62d5-4a6f-ba47-242712b43d0a-kube-api-access-b9nh7\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:36.000121 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.000116    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862bf26d-62d5-4a6f-ba47-242712b43d0a-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:36.024906 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.024868    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:36.184995 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.184962    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt"]
Oct 30 23:04:36.188162 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.188127    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:36.189551 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.189532    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bff950-67c96cc479-ll5pt"]
Oct 30 23:04:36.364218 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.364181    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:36.533080 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.533023    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main" containerID="cri-o://1e92b74aeb385c44c95f1d7b3ff4f3cf0ffce40a856436a3bf727e73d4b7050e" gracePeriod=45
Oct 30 23:04:36.660260 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.660208    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb"]
Oct 30 23:04:36.694486 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.694447    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48"]
Oct 30 23:04:36.764271 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.757674    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:36.778698 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.778664    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="862bf26d-62d5-4a6f-ba47-242712b43d0a" path="/var/lib/kubelet/pods/862bf26d-62d5-4a6f-ba47-242712b43d0a/volumes"
Oct 30 23:04:36.779257 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.779237    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048" path="/var/lib/kubelet/pods/bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048/volumes"
Oct 30 23:04:36.806176 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.806153    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 23:04:36.842234 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.842191    4889 generic.go:356] "Generic (PLEG): container finished" podID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerID="1e92b74aeb385c44c95f1d7b3ff4f3cf0ffce40a856436a3bf727e73d4b7050e" exitCode=0
Oct 30 23:04:36.842402 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.842256    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"
Oct 30 23:04:36.842402 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.842271    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" event={"ID":"da44d50b-144e-4f4a-bd5b-195b78448b0f","Type":"ContainerDied","Data":"1e92b74aeb385c44c95f1d7b3ff4f3cf0ffce40a856436a3bf727e73d4b7050e"}
Oct 30 23:04:36.842402 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.842309    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf" event={"ID":"da44d50b-144e-4f4a-bd5b-195b78448b0f","Type":"ContainerDied","Data":"9bdd22b29aa43e2eb86eb55a6bd1f6e2000bfbece2e7c6eeeefd66d9bee26527"}
Oct 30 23:04:36.842402 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.842322    4889 scope.go:122] "RemoveContainer" containerID="1e92b74aeb385c44c95f1d7b3ff4f3cf0ffce40a856436a3bf727e73d4b7050e"
Oct 30 23:04:36.854253 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.854215    4889 scope.go:122] "RemoveContainer" containerID="1e92b74aeb385c44c95f1d7b3ff4f3cf0ffce40a856436a3bf727e73d4b7050e"
Oct 30 23:04:36.854735 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:04:36.854707    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e92b74aeb385c44c95f1d7b3ff4f3cf0ffce40a856436a3bf727e73d4b7050e\": container with ID starting with 1e92b74aeb385c44c95f1d7b3ff4f3cf0ffce40a856436a3bf727e73d4b7050e not found: ID does not exist" containerID="1e92b74aeb385c44c95f1d7b3ff4f3cf0ffce40a856436a3bf727e73d4b7050e"
Oct 30 23:04:36.854842 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.854746    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e92b74aeb385c44c95f1d7b3ff4f3cf0ffce40a856436a3bf727e73d4b7050e"} err="failed to get container status \"1e92b74aeb385c44c95f1d7b3ff4f3cf0ffce40a856436a3bf727e73d4b7050e\": rpc error: code = NotFound desc = could not find container \"1e92b74aeb385c44c95f1d7b3ff4f3cf0ffce40a856436a3bf727e73d4b7050e\": container with ID starting with 1e92b74aeb385c44c95f1d7b3ff4f3cf0ffce40a856436a3bf727e73d4b7050e not found: ID does not exist"
Oct 30 23:04:36.909552 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.909510    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-tmp-dir\") pod \"da44d50b-144e-4f4a-bd5b-195b78448b0f\" (UID: \"da44d50b-144e-4f4a-bd5b-195b78448b0f\") "
Oct 30 23:04:36.909750 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.909569    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-dshm\") pod \"da44d50b-144e-4f4a-bd5b-195b78448b0f\" (UID: \"da44d50b-144e-4f4a-bd5b-195b78448b0f\") "
Oct 30 23:04:36.909750 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.909634    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da44d50b-144e-4f4a-bd5b-195b78448b0f-tls-certs\" (UniqueName: \"kubernetes.io/secret/da44d50b-144e-4f4a-bd5b-195b78448b0f-tls-certs\") pod \"da44d50b-144e-4f4a-bd5b-195b78448b0f\" (UID: \"da44d50b-144e-4f4a-bd5b-195b78448b0f\") "
Oct 30 23:04:36.909750 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.909666    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da44d50b-144e-4f4a-bd5b-195b78448b0f-kube-api-access-swwfs\" (UniqueName: \"kubernetes.io/projected/da44d50b-144e-4f4a-bd5b-195b78448b0f-kube-api-access-swwfs\") pod \"da44d50b-144e-4f4a-bd5b-195b78448b0f\" (UID: \"da44d50b-144e-4f4a-bd5b-195b78448b0f\") "
Oct 30 23:04:36.909750 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.909694    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-home\" (UniqueName: \"kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-home\") pod \"da44d50b-144e-4f4a-bd5b-195b78448b0f\" (UID: \"da44d50b-144e-4f4a-bd5b-195b78448b0f\") "
Oct 30 23:04:36.910026 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.909760    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-model-cache\") pod \"da44d50b-144e-4f4a-bd5b-195b78448b0f\" (UID: \"da44d50b-144e-4f4a-bd5b-195b78448b0f\") "
Oct 30 23:04:36.910026 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.909846    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-tmp-dir" pod "da44d50b-144e-4f4a-bd5b-195b78448b0f" (UID: "da44d50b-144e-4f4a-bd5b-195b78448b0f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:36.910026 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.910001    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-home" pod "da44d50b-144e-4f4a-bd5b-195b78448b0f" (UID: "da44d50b-144e-4f4a-bd5b-195b78448b0f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:36.910151 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.910133    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-model-cache" pod "da44d50b-144e-4f4a-bd5b-195b78448b0f" (UID: "da44d50b-144e-4f4a-bd5b-195b78448b0f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:36.910210 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.910193    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:36.910285 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.910217    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:36.912099 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.912076    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da44d50b-144e-4f4a-bd5b-195b78448b0f-tls-certs" pod "da44d50b-144e-4f4a-bd5b-195b78448b0f" (UID: "da44d50b-144e-4f4a-bd5b-195b78448b0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:36.912419 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.912399    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-dshm" pod "da44d50b-144e-4f4a-bd5b-195b78448b0f" (UID: "da44d50b-144e-4f4a-bd5b-195b78448b0f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:36.912478 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.912425    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da44d50b-144e-4f4a-bd5b-195b78448b0f-kube-api-access-swwfs" pod "da44d50b-144e-4f4a-bd5b-195b78448b0f" (UID: "da44d50b-144e-4f4a-bd5b-195b78448b0f"). InnerVolumeSpecName "kube-api-access-swwfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:36.920896 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:36.920841    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:37.011537 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:37.011506    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:37.011537 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:37.011533    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da44d50b-144e-4f4a-bd5b-195b78448b0f-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:37.011799 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:37.011548    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swwfs\" (UniqueName: \"kubernetes.io/projected/da44d50b-144e-4f4a-bd5b-195b78448b0f-kube-api-access-swwfs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:37.011799 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:37.011557    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da44d50b-144e-4f4a-bd5b-195b78448b0f-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:37.190603 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:37.190569    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"]
Oct 30 23:04:37.204276 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:37.204241    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-421583/distinct-2-w2-421583-kserve-565d4d7cc-vlllf"]
Oct 30 23:04:38.122571 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:38.122529    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:38.773417 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:38.773387    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" path="/var/lib/kubelet/pods/da44d50b-144e-4f4a-bd5b-195b78448b0f/volumes"
Oct 30 23:04:39.905926 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:39.905883    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:40.301589 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:04:40.301554    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:04:41.301650 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.301612    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5"]
Oct 30 23:04:41.302211 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.301881    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5" podUID="cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad" containerName="maas-api" containerID="cri-o://c1538ffd80e4348aeb7b5a85fad0c1188983302480493c923d35b05137a4d21c" gracePeriod=30
Oct 30 23:04:41.417864 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.417824    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:41.565891 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.565808    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5"
Oct 30 23:04:41.652828 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.652788    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-openshift-service-ca\") pod \"cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad\" (UID: \"cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad\") "
Oct 30 23:04:41.653023 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.652847    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-metrics-certs\") pod \"cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad\" (UID: \"cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad\") "
Oct 30 23:04:41.653023 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.652901    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-maas-api-tls\") pod \"cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad\" (UID: \"cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad\") "
Oct 30 23:04:41.653023 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.652953    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-kube-api-access-x2nwt\" (UniqueName: \"kubernetes.io/projected/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-kube-api-access-x2nwt\") pod \"cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad\" (UID: \"cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad\") "
Oct 30 23:04:41.653216 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.653155    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-openshift-service-ca" pod "cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad" (UID: "cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:41.653435 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.653410    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:41.655274 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.655218    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-maas-api-tls" pod "cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad" (UID: "cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:41.655274 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.655259    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-metrics-certs" pod "cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad" (UID: "cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:41.655439 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.655285    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-kube-api-access-x2nwt" pod "cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad" (UID: "cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad"). InnerVolumeSpecName "kube-api-access-x2nwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:41.661047 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.661009    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb" podUID="6d033fb5-f7fb-4480-9cf6-57effbff1402" containerName="payload-processing" containerID="cri-o://0fcf7dfcbcdad3634ad5e3b42330135b9db8e1bb7855f274d513b8e4306747ab" gracePeriod=25
Oct 30 23:04:41.695136 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.695067    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48" podUID="51bd7538-efdc-40b3-b43c-35e7714a1014" containerName="payload-pre-processing" containerID="cri-o://978f57ae7cfbaf7ea8ec4b8953ba32fb54ef7f7fc4e3f3cef67ed1207689e463" gracePeriod=25
Oct 30 23:04:41.754592 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.754556    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:41.754592 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.754587    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:41.754808 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.754618    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2nwt\" (UniqueName: \"kubernetes.io/projected/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad-kube-api-access-x2nwt\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:41.880262 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.876610    4889 generic.go:356] "Generic (PLEG): container finished" podID="51bd7538-efdc-40b3-b43c-35e7714a1014" containerID="978f57ae7cfbaf7ea8ec4b8953ba32fb54ef7f7fc4e3f3cef67ed1207689e463" exitCode=0
Oct 30 23:04:41.880262 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.876734    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48" event={"ID":"51bd7538-efdc-40b3-b43c-35e7714a1014","Type":"ContainerDied","Data":"978f57ae7cfbaf7ea8ec4b8953ba32fb54ef7f7fc4e3f3cef67ed1207689e463"}
Oct 30 23:04:41.880262 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.878679    4889 generic.go:356] "Generic (PLEG): container finished" podID="cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad" containerID="c1538ffd80e4348aeb7b5a85fad0c1188983302480493c923d35b05137a4d21c" exitCode=0
Oct 30 23:04:41.880262 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.878745    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5"
Oct 30 23:04:41.880262 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.878824    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5" event={"ID":"cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad","Type":"ContainerDied","Data":"c1538ffd80e4348aeb7b5a85fad0c1188983302480493c923d35b05137a4d21c"}
Oct 30 23:04:41.880262 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.878856    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5" event={"ID":"cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad","Type":"ContainerDied","Data":"e81c61fa4c566680639181e46b4616dd1ce42466a7f6ca4fa00315ccb8fba6bf"}
Oct 30 23:04:41.880262 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.878877    4889 scope.go:122] "RemoveContainer" containerID="c1538ffd80e4348aeb7b5a85fad0c1188983302480493c923d35b05137a4d21c"
Oct 30 23:04:41.883260 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.881857    4889 generic.go:356] "Generic (PLEG): container finished" podID="6d033fb5-f7fb-4480-9cf6-57effbff1402" containerID="0fcf7dfcbcdad3634ad5e3b42330135b9db8e1bb7855f274d513b8e4306747ab" exitCode=0
Oct 30 23:04:41.883260 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.881955    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb" event={"ID":"6d033fb5-f7fb-4480-9cf6-57effbff1402","Type":"ContainerDied","Data":"0fcf7dfcbcdad3634ad5e3b42330135b9db8e1bb7855f274d513b8e4306747ab"}
Oct 30 23:04:41.898116 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.897867    4889 scope.go:122] "RemoveContainer" containerID="c1538ffd80e4348aeb7b5a85fad0c1188983302480493c923d35b05137a4d21c"
Oct 30 23:04:41.899471 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:04:41.899441    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1538ffd80e4348aeb7b5a85fad0c1188983302480493c923d35b05137a4d21c\": container with ID starting with c1538ffd80e4348aeb7b5a85fad0c1188983302480493c923d35b05137a4d21c not found: ID does not exist" containerID="c1538ffd80e4348aeb7b5a85fad0c1188983302480493c923d35b05137a4d21c"
Oct 30 23:04:41.899609 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.899482    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1538ffd80e4348aeb7b5a85fad0c1188983302480493c923d35b05137a4d21c"} err="failed to get container status \"c1538ffd80e4348aeb7b5a85fad0c1188983302480493c923d35b05137a4d21c\": rpc error: code = NotFound desc = could not find container \"c1538ffd80e4348aeb7b5a85fad0c1188983302480493c923d35b05137a4d21c\": container with ID starting with c1538ffd80e4348aeb7b5a85fad0c1188983302480493c923d35b05137a4d21c not found: ID does not exist"
Oct 30 23:04:41.936893 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.936865    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb"
Oct 30 23:04:41.945089 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.945061    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5"]
Oct 30 23:04:41.948745 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.948717    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-421583-c958dc6b-pjxh5"]
Oct 30 23:04:41.956473 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.956447    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d033fb5-f7fb-4480-9cf6-57effbff1402-kube-api-access-grjtr\" (UniqueName: \"kubernetes.io/projected/6d033fb5-f7fb-4480-9cf6-57effbff1402-kube-api-access-grjtr\") pod \"6d033fb5-f7fb-4480-9cf6-57effbff1402\" (UID: \"6d033fb5-f7fb-4480-9cf6-57effbff1402\") "
Oct 30 23:04:41.956685 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.956648    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d033fb5-f7fb-4480-9cf6-57effbff1402-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d033fb5-f7fb-4480-9cf6-57effbff1402-plugins-config-volume\") pod \"6d033fb5-f7fb-4480-9cf6-57effbff1402\" (UID: \"6d033fb5-f7fb-4480-9cf6-57effbff1402\") "
Oct 30 23:04:41.957009 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.956988    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d033fb5-f7fb-4480-9cf6-57effbff1402-plugins-config-volume" pod "6d033fb5-f7fb-4480-9cf6-57effbff1402" (UID: "6d033fb5-f7fb-4480-9cf6-57effbff1402"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:41.959855 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.959827    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d033fb5-f7fb-4480-9cf6-57effbff1402-kube-api-access-grjtr" pod "6d033fb5-f7fb-4480-9cf6-57effbff1402" (UID: "6d033fb5-f7fb-4480-9cf6-57effbff1402"). InnerVolumeSpecName "kube-api-access-grjtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:41.960296 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.960272    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:41.978244 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:41.978207    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48"
Oct 30 23:04:42.058161 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:42.058126    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51bd7538-efdc-40b3-b43c-35e7714a1014-kube-api-access-cm94j\" (UniqueName: \"kubernetes.io/projected/51bd7538-efdc-40b3-b43c-35e7714a1014-kube-api-access-cm94j\") pod \"51bd7538-efdc-40b3-b43c-35e7714a1014\" (UID: \"51bd7538-efdc-40b3-b43c-35e7714a1014\") "
Oct 30 23:04:42.058374 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:42.058279    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51bd7538-efdc-40b3-b43c-35e7714a1014-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51bd7538-efdc-40b3-b43c-35e7714a1014-plugins-config-volume\") pod \"51bd7538-efdc-40b3-b43c-35e7714a1014\" (UID: \"51bd7538-efdc-40b3-b43c-35e7714a1014\") "
Oct 30 23:04:42.058607 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:42.058591    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d033fb5-f7fb-4480-9cf6-57effbff1402-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:42.058679 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:42.058611    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grjtr\" (UniqueName: \"kubernetes.io/projected/6d033fb5-f7fb-4480-9cf6-57effbff1402-kube-api-access-grjtr\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:42.058735 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:42.058709    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51bd7538-efdc-40b3-b43c-35e7714a1014-plugins-config-volume" pod "51bd7538-efdc-40b3-b43c-35e7714a1014" (UID: "51bd7538-efdc-40b3-b43c-35e7714a1014"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:42.060340 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:42.060317    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51bd7538-efdc-40b3-b43c-35e7714a1014-kube-api-access-cm94j" pod "51bd7538-efdc-40b3-b43c-35e7714a1014" (UID: "51bd7538-efdc-40b3-b43c-35e7714a1014"). InnerVolumeSpecName "kube-api-access-cm94j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:42.132183 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:42.132076    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:42.159634 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:42.159596    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm94j\" (UniqueName: \"kubernetes.io/projected/51bd7538-efdc-40b3-b43c-35e7714a1014-kube-api-access-cm94j\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:42.159634 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:42.159627    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51bd7538-efdc-40b3-b43c-35e7714a1014-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:42.540427 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:42.540379    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:42.774283 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:42.774217    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad" path="/var/lib/kubelet/pods/cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad/volumes"
Oct 30 23:04:42.889148 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:42.889044    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48"
Oct 30 23:04:42.889148 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:42.889048    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48" event={"ID":"51bd7538-efdc-40b3-b43c-35e7714a1014","Type":"ContainerDied","Data":"cdde1b2dc9d180d09ce553561fda22db61ad8d83282f206a4570b20810793481"}
Oct 30 23:04:42.889456 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:42.889180    4889 scope.go:122] "RemoveContainer" containerID="978f57ae7cfbaf7ea8ec4b8953ba32fb54ef7f7fc4e3f3cef67ed1207689e463"
Oct 30 23:04:42.892403 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:42.892366    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb" event={"ID":"6d033fb5-f7fb-4480-9cf6-57effbff1402","Type":"ContainerDied","Data":"49d90ca08d9dc563bf71f8d0862d3e3805712f4acff0b85e2d59ef002da3be77"}
Oct 30 23:04:42.892403 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:42.892390    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb"
Oct 30 23:04:42.899683 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:42.899662    4889 scope.go:122] "RemoveContainer" containerID="0fcf7dfcbcdad3634ad5e3b42330135b9db8e1bb7855f274d513b8e4306747ab"
Oct 30 23:04:42.943711 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:42.943675    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48"]
Oct 30 23:04:42.957802 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:42.957760    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-09d0d5-75f8f97f55-6hq48"]
Oct 30 23:04:42.994830 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:42.994799    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb"]
Oct 30 23:04:42.998888 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:42.998863    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-09d0d5-85cb95d7ff-n4lzb"]
Oct 30 23:04:43.516531 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.516499    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:43.620546 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.620508    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51c10141-5pjp6"]
Oct 30 23:04:43.620957 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.620947    4889 cpu_manager.go:414] "containerMap: removing container" podUID="df4966ae-49e6-461e-b677-6111da780e93" containerName="istio-proxy"
Oct 30 23:04:43.621002 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.620960    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df4966ae-49e6-461e-b677-6111da780e93" containerName="istio-proxy"
Oct 30 23:04:43.621002 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.620973    4889 cpu_manager.go:414] "containerMap: removing container" podUID="cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad" containerName="maas-api"
Oct 30 23:04:43.621002 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.620980    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad" containerName="maas-api"
Oct 30 23:04:43.621002 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.620991    4889 cpu_manager.go:414] "containerMap: removing container" podUID="862bf26d-62d5-4a6f-ba47-242712b43d0a" containerName="payload-pre-processing"
Oct 30 23:04:43.621002 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.620996    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="862bf26d-62d5-4a6f-ba47-242712b43d0a" containerName="payload-pre-processing"
Oct 30 23:04:43.621002 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621003    4889 cpu_manager.go:414] "containerMap: removing container" podUID="341cd65b-6e79-41b5-996c-1b866e57780c" containerName="payload-processing"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621009    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="341cd65b-6e79-41b5-996c-1b866e57780c" containerName="payload-processing"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621018    4889 cpu_manager.go:414] "containerMap: removing container" podUID="9833755d-9e30-41f6-806f-688a9cb72349" containerName="istio-proxy"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621024    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9833755d-9e30-41f6-806f-688a9cb72349" containerName="istio-proxy"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621030    4889 cpu_manager.go:414] "containerMap: removing container" podUID="65b844a4-133d-40a6-bc49-81bbffc94221" containerName="istio-proxy"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621034    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b844a4-133d-40a6-bc49-81bbffc94221" containerName="istio-proxy"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621042    4889 cpu_manager.go:414] "containerMap: removing container" podUID="d7ce41e7-a1cb-4f49-a247-da6c57d34d53" containerName="istio-proxy"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621047    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7ce41e7-a1cb-4f49-a247-da6c57d34d53" containerName="istio-proxy"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621055    4889 cpu_manager.go:414] "containerMap: removing container" podUID="df4966ae-49e6-461e-b677-6111da780e93" containerName="istio-proxy"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621059    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df4966ae-49e6-461e-b677-6111da780e93" containerName="istio-proxy"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621067    4889 cpu_manager.go:414] "containerMap: removing container" podUID="dafe75fd-77ad-49d3-aafb-35f28fcb2954" containerName="payload-pre-processing"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621072    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dafe75fd-77ad-49d3-aafb-35f28fcb2954" containerName="payload-pre-processing"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621080    4889 cpu_manager.go:414] "containerMap: removing container" podUID="24e8fc0c-97b0-47f1-ba4b-5ec430718d31" containerName="istio-proxy"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621085    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24e8fc0c-97b0-47f1-ba4b-5ec430718d31" containerName="istio-proxy"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621092    4889 cpu_manager.go:414] "containerMap: removing container" podUID="c02298a7-464d-4cc4-8bef-3be1dfc6f3ab" containerName="istio-proxy"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621096    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c02298a7-464d-4cc4-8bef-3be1dfc6f3ab" containerName="istio-proxy"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621102    4889 cpu_manager.go:414] "containerMap: removing container" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621107    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621118    4889 cpu_manager.go:414] "containerMap: removing container" podUID="51bd7538-efdc-40b3-b43c-35e7714a1014" containerName="payload-pre-processing"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621123    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51bd7538-efdc-40b3-b43c-35e7714a1014" containerName="payload-pre-processing"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621131    4889 cpu_manager.go:414] "containerMap: removing container" podUID="b37648ec-d503-43af-aef6-ec9da60a0902" containerName="payload-pre-processing"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621136    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b37648ec-d503-43af-aef6-ec9da60a0902" containerName="payload-pre-processing"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621142    4889 cpu_manager.go:414] "containerMap: removing container" podUID="bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048" containerName="payload-processing"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621146    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048" containerName="payload-processing"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621155    4889 cpu_manager.go:414] "containerMap: removing container" podUID="6d033fb5-f7fb-4480-9cf6-57effbff1402" containerName="payload-processing"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621160    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d033fb5-f7fb-4480-9cf6-57effbff1402" containerName="payload-processing"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621167    4889 cpu_manager.go:414] "containerMap: removing container" podUID="cc0df870-2a4a-404b-9678-bdc054dce3e3" containerName="payload-pre-processing"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621171    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0df870-2a4a-404b-9678-bdc054dce3e3" containerName="payload-pre-processing"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621178    4889 cpu_manager.go:414] "containerMap: removing container" podUID="2de71837-99a5-4e01-ae12-385e722759eb" containerName="payload-pre-processing"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621182    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2de71837-99a5-4e01-ae12-385e722759eb" containerName="payload-pre-processing"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621189    4889 cpu_manager.go:414] "containerMap: removing container" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621194    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621203    4889 cpu_manager.go:414] "containerMap: removing container" podUID="d131b55f-ea6b-4f31-8053-fe81f0c0527e" containerName="payload-pre-processing"
Oct 30 23:04:43.621194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621208    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d131b55f-ea6b-4f31-8053-fe81f0c0527e" containerName="payload-pre-processing"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621214    4889 cpu_manager.go:414] "containerMap: removing container" podUID="428b38e7-52ac-4f32-a8be-ea74f109c4d1" containerName="payload-processing"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621248    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="428b38e7-52ac-4f32-a8be-ea74f109c4d1" containerName="payload-processing"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621254    4889 cpu_manager.go:414] "containerMap: removing container" podUID="b812174d-906c-487b-91a9-7a1620aca44e" containerName="payload-processing"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621259    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b812174d-906c-487b-91a9-7a1620aca44e" containerName="payload-processing"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621265    4889 cpu_manager.go:414] "containerMap: removing container" podUID="ac794af8-6cd6-473f-bee9-3ea457729889" containerName="maas-api"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621269    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac794af8-6cd6-473f-bee9-3ea457729889" containerName="maas-api"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621277    4889 cpu_manager.go:414] "containerMap: removing container" podUID="ddfac372-f0f0-4e7b-a661-007adc297ebd" containerName="maas-api"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621282    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddfac372-f0f0-4e7b-a661-007adc297ebd" containerName="maas-api"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621287    4889 cpu_manager.go:414] "containerMap: removing container" podUID="368f6471-b984-4935-92e6-047c8f81053f" containerName="payload-processing"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621292    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="368f6471-b984-4935-92e6-047c8f81053f" containerName="payload-processing"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621299    4889 cpu_manager.go:414] "containerMap: removing container" podUID="664cb2dc-09fb-45ca-b315-7ab915110da7" containerName="maas-api"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621304    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="664cb2dc-09fb-45ca-b315-7ab915110da7" containerName="maas-api"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621310    4889 cpu_manager.go:414] "containerMap: removing container" podUID="2e0a2179-dbbd-49c3-b6af-90aec117a44c" containerName="payload-pre-processing"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621314    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e0a2179-dbbd-49c3-b6af-90aec117a44c" containerName="payload-pre-processing"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621324    4889 cpu_manager.go:414] "containerMap: removing container" podUID="a5031ee6-029b-44a4-8c97-68a38a1d12d4" containerName="payload-processing"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621328    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5031ee6-029b-44a4-8c97-68a38a1d12d4" containerName="payload-processing"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621391    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="b812174d-906c-487b-91a9-7a1620aca44e" containerName="payload-processing"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621401    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac794af8-6cd6-473f-bee9-3ea457729889" containerName="maas-api"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621408    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddfac372-f0f0-4e7b-a661-007adc297ebd" containerName="maas-api"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621413    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="c02298a7-464d-4cc4-8bef-3be1dfc6f3ab" containerName="istio-proxy"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621419    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="368f6471-b984-4935-92e6-047c8f81053f" containerName="payload-processing"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621425    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7ce41e7-a1cb-4f49-a247-da6c57d34d53" containerName="istio-proxy"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621429    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="664cb2dc-09fb-45ca-b315-7ab915110da7" containerName="maas-api"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621435    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e0a2179-dbbd-49c3-b6af-90aec117a44c" containerName="payload-pre-processing"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621441    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5031ee6-029b-44a4-8c97-68a38a1d12d4" containerName="payload-processing"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621448    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="df4966ae-49e6-461e-b677-6111da780e93" containerName="istio-proxy"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621453    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdb1b2e2-ed49-4662-b1e0-07d1e1cce6ad" containerName="maas-api"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621458    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="862bf26d-62d5-4a6f-ba47-242712b43d0a" containerName="payload-pre-processing"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621462    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="341cd65b-6e79-41b5-996c-1b866e57780c" containerName="payload-processing"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621467    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="9833755d-9e30-41f6-806f-688a9cb72349" containerName="istio-proxy"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621472    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="65b844a4-133d-40a6-bc49-81bbffc94221" containerName="istio-proxy"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621477    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7ce41e7-a1cb-4f49-a247-da6c57d34d53" containerName="istio-proxy"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621483    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="df4966ae-49e6-461e-b677-6111da780e93" containerName="istio-proxy"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621489    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="dafe75fd-77ad-49d3-aafb-35f28fcb2954" containerName="payload-pre-processing"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621494    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="24e8fc0c-97b0-47f1-ba4b-5ec430718d31" containerName="istio-proxy"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621499    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd0ddc98-49ec-4ca9-ba5d-1db4b785b186" containerName="main"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621505    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="51bd7538-efdc-40b3-b43c-35e7714a1014" containerName="payload-pre-processing"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621511    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="9833755d-9e30-41f6-806f-688a9cb72349" containerName="istio-proxy"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621518    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="b37648ec-d503-43af-aef6-ec9da60a0902" containerName="payload-pre-processing"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621523    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7bf3b1-7a2a-4bc6-82d4-9a28c193c048" containerName="payload-processing"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621529    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d033fb5-f7fb-4480-9cf6-57effbff1402" containerName="payload-processing"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621534    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc0df870-2a4a-404b-9678-bdc054dce3e3" containerName="payload-pre-processing"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621539    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="2de71837-99a5-4e01-ae12-385e722759eb" containerName="payload-pre-processing"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621544    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="da44d50b-144e-4f4a-bd5b-195b78448b0f" containerName="main"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621549    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="d131b55f-ea6b-4f31-8053-fe81f0c0527e" containerName="payload-pre-processing"
Oct 30 23:04:43.622240 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.621554    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="428b38e7-52ac-4f32-a8be-ea74f109c4d1" containerName="payload-processing"
Oct 30 23:04:43.626111 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.626092    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c10141-5pjp6"
Oct 30 23:04:43.642308 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.641458    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51c10141-5pjp6"]
Oct 30 23:04:43.651915 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.651891    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:04:43.652058 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.652015    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:04:43.675260 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.675214    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d893ba75-1d6a-404b-be7d-8b78ca3e6c62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51c10141-5pjp6\" (UID: \"d893ba75-1d6a-404b-be7d-8b78ca3e6c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c10141-5pjp6"
Oct 30 23:04:43.675458 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.675277    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d893ba75-1d6a-404b-be7d-8b78ca3e6c62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51c10141-5pjp6\" (UID: \"d893ba75-1d6a-404b-be7d-8b78ca3e6c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c10141-5pjp6"
Oct 30 23:04:43.675458 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.675387    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bsz2\" (UniqueName: \"kubernetes.io/projected/d893ba75-1d6a-404b-be7d-8b78ca3e6c62-kube-api-access-2bsz2\") pod \"maas-api-revoke-sub-51c10141-5pjp6\" (UID: \"d893ba75-1d6a-404b-be7d-8b78ca3e6c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c10141-5pjp6"
Oct 30 23:04:43.702202 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.702166    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:43.776761 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.776651    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d893ba75-1d6a-404b-be7d-8b78ca3e6c62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51c10141-5pjp6\" (UID: \"d893ba75-1d6a-404b-be7d-8b78ca3e6c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c10141-5pjp6"
Oct 30 23:04:43.776761 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.776739    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d893ba75-1d6a-404b-be7d-8b78ca3e6c62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51c10141-5pjp6\" (UID: \"d893ba75-1d6a-404b-be7d-8b78ca3e6c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c10141-5pjp6"
Oct 30 23:04:43.776982 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.776787    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bsz2\" (UniqueName: \"kubernetes.io/projected/d893ba75-1d6a-404b-be7d-8b78ca3e6c62-kube-api-access-2bsz2\") pod \"maas-api-revoke-sub-51c10141-5pjp6\" (UID: \"d893ba75-1d6a-404b-be7d-8b78ca3e6c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c10141-5pjp6"
Oct 30 23:04:43.777464 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.777439    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d893ba75-1d6a-404b-be7d-8b78ca3e6c62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51c10141-5pjp6\" (UID: \"d893ba75-1d6a-404b-be7d-8b78ca3e6c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c10141-5pjp6"
Oct 30 23:04:43.779359 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.779340    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d893ba75-1d6a-404b-be7d-8b78ca3e6c62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51c10141-5pjp6\" (UID: \"d893ba75-1d6a-404b-be7d-8b78ca3e6c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c10141-5pjp6"
Oct 30 23:04:43.797899 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.797872    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bsz2\" (UniqueName: \"kubernetes.io/projected/d893ba75-1d6a-404b-be7d-8b78ca3e6c62-kube-api-access-2bsz2\") pod \"maas-api-revoke-sub-51c10141-5pjp6\" (UID: \"d893ba75-1d6a-404b-be7d-8b78ca3e6c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c10141-5pjp6"
Oct 30 23:04:43.878425 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.878381    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:43.941964 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:43.941909    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c10141-5pjp6"
Oct 30 23:04:44.126460 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.126366    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:44.253818 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.252602    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51c10141-5pjp6"]
Oct 30 23:04:44.253818 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:04:44.253237    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd893ba75_1d6a_404b_be7d_8b78ca3e6c62.slice/crio-712cf55d51303cea8bf918735389c6b8895ea6879f1b0e9a5e56b30b5429ea14 WatchSource:0}: Error finding container 712cf55d51303cea8bf918735389c6b8895ea6879f1b0e9a5e56b30b5429ea14: Status 404 returned error can't find the container with id 712cf55d51303cea8bf918735389c6b8895ea6879f1b0e9a5e56b30b5429ea14
Oct 30 23:04:44.350611 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.350572    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq"]
Oct 30 23:04:44.351021 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.351009    4889 cpu_manager.go:414] "containerMap: removing container" podUID="9833755d-9e30-41f6-806f-688a9cb72349" containerName="istio-proxy"
Oct 30 23:04:44.351102 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.351023    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9833755d-9e30-41f6-806f-688a9cb72349" containerName="istio-proxy"
Oct 30 23:04:44.351102 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.351034    4889 cpu_manager.go:414] "containerMap: removing container" podUID="d7ce41e7-a1cb-4f49-a247-da6c57d34d53" containerName="istio-proxy"
Oct 30 23:04:44.351102 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.351039    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7ce41e7-a1cb-4f49-a247-da6c57d34d53" containerName="istio-proxy"
Oct 30 23:04:44.351102 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.351052    4889 cpu_manager.go:414] "containerMap: removing container" podUID="c02298a7-464d-4cc4-8bef-3be1dfc6f3ab" containerName="istio-proxy"
Oct 30 23:04:44.351102 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.351059    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c02298a7-464d-4cc4-8bef-3be1dfc6f3ab" containerName="istio-proxy"
Oct 30 23:04:44.351308 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.351129    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="c02298a7-464d-4cc4-8bef-3be1dfc6f3ab" containerName="istio-proxy"
Oct 30 23:04:44.352792 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.352764    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main" containerID="cri-o://ea23841308d9459ab2e480beee95231491f95a06b2621d29dd2489c61ccca6c2" gracePeriod=45
Oct 30 23:04:44.355902 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.355880    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq"
Oct 30 23:04:44.367740 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.367703    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main" containerID="cri-o://d67e46092f17279a11459e9f4ef2bfffd7e07d98886da23307f1410590683891" gracePeriod=45
Oct 30 23:04:44.382765 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.382639    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq"]
Oct 30 23:04:44.389133 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.387723    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5rvn\" (UniqueName: \"kubernetes.io/projected/4c8f8098-b597-45ce-ac4b-60aa0910e60a-kube-api-access-r5rvn\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq\" (UID: \"4c8f8098-b597-45ce-ac4b-60aa0910e60a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq"
Oct 30 23:04:44.389133 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.387807    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8f8098-b597-45ce-ac4b-60aa0910e60a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq\" (UID: \"4c8f8098-b597-45ce-ac4b-60aa0910e60a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq"
Oct 30 23:04:44.389133 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.387844    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8f8098-b597-45ce-ac4b-60aa0910e60a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq\" (UID: \"4c8f8098-b597-45ce-ac4b-60aa0910e60a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq"
Oct 30 23:04:44.488936 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.488901    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8f8098-b597-45ce-ac4b-60aa0910e60a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq\" (UID: \"4c8f8098-b597-45ce-ac4b-60aa0910e60a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq"
Oct 30 23:04:44.489080 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.488948    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8f8098-b597-45ce-ac4b-60aa0910e60a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq\" (UID: \"4c8f8098-b597-45ce-ac4b-60aa0910e60a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq"
Oct 30 23:04:44.489146 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.489090    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5rvn\" (UniqueName: \"kubernetes.io/projected/4c8f8098-b597-45ce-ac4b-60aa0910e60a-kube-api-access-r5rvn\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq\" (UID: \"4c8f8098-b597-45ce-ac4b-60aa0910e60a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq"
Oct 30 23:04:44.489853 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.489820    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8f8098-b597-45ce-ac4b-60aa0910e60a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq\" (UID: \"4c8f8098-b597-45ce-ac4b-60aa0910e60a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq"
Oct 30 23:04:44.491838 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.491811    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8f8098-b597-45ce-ac4b-60aa0910e60a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq\" (UID: \"4c8f8098-b597-45ce-ac4b-60aa0910e60a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq"
Oct 30 23:04:44.514430 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.514392    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5rvn\" (UniqueName: \"kubernetes.io/projected/4c8f8098-b597-45ce-ac4b-60aa0910e60a-kube-api-access-r5rvn\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq\" (UID: \"4c8f8098-b597-45ce-ac4b-60aa0910e60a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq"
Oct 30 23:04:44.614966 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.614937    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 23:04:44.649949 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.649920    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 23:04:44.671445 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.671407    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq"
Oct 30 23:04:44.691533 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.691495    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-model-cache\") pod \"0e99cfe5-3268-401d-b985-7634e2bac80b\" (UID: \"0e99cfe5-3268-401d-b985-7634e2bac80b\") "
Oct 30 23:04:44.692016 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.691551    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-home\" (UniqueName: \"kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-home\") pod \"0e99cfe5-3268-401d-b985-7634e2bac80b\" (UID: \"0e99cfe5-3268-401d-b985-7634e2bac80b\") "
Oct 30 23:04:44.692016 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.691615    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d18c6ee-2f7b-4a85-9065-6c925004973d-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d18c6ee-2f7b-4a85-9065-6c925004973d-tls-certs\") pod \"5d18c6ee-2f7b-4a85-9065-6c925004973d\" (UID: \"5d18c6ee-2f7b-4a85-9065-6c925004973d\") "
Oct 30 23:04:44.692016 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.691672    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-tmp-dir\") pod \"5d18c6ee-2f7b-4a85-9065-6c925004973d\" (UID: \"5d18c6ee-2f7b-4a85-9065-6c925004973d\") "
Oct 30 23:04:44.692016 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.691703    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-model-cache\") pod \"5d18c6ee-2f7b-4a85-9065-6c925004973d\" (UID: \"5d18c6ee-2f7b-4a85-9065-6c925004973d\") "
Oct 30 23:04:44.692016 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.691747    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-home\" (UniqueName: \"kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-home\") pod \"5d18c6ee-2f7b-4a85-9065-6c925004973d\" (UID: \"5d18c6ee-2f7b-4a85-9065-6c925004973d\") "
Oct 30 23:04:44.692016 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.691779    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-tmp-dir\") pod \"0e99cfe5-3268-401d-b985-7634e2bac80b\" (UID: \"0e99cfe5-3268-401d-b985-7634e2bac80b\") "
Oct 30 23:04:44.692016 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.691840    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-dshm\") pod \"0e99cfe5-3268-401d-b985-7634e2bac80b\" (UID: \"0e99cfe5-3268-401d-b985-7634e2bac80b\") "
Oct 30 23:04:44.692016 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.691890    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d18c6ee-2f7b-4a85-9065-6c925004973d-kube-api-access-5lc5g\" (UniqueName: \"kubernetes.io/projected/5d18c6ee-2f7b-4a85-9065-6c925004973d-kube-api-access-5lc5g\") pod \"5d18c6ee-2f7b-4a85-9065-6c925004973d\" (UID: \"5d18c6ee-2f7b-4a85-9065-6c925004973d\") "
Oct 30 23:04:44.692016 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.691939    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-home" pod "0e99cfe5-3268-401d-b985-7634e2bac80b" (UID: "0e99cfe5-3268-401d-b985-7634e2bac80b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:44.692016 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.691958    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-dshm\") pod \"5d18c6ee-2f7b-4a85-9065-6c925004973d\" (UID: \"5d18c6ee-2f7b-4a85-9065-6c925004973d\") "
Oct 30 23:04:44.693363 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.692047    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e99cfe5-3268-401d-b985-7634e2bac80b-tls-certs\" (UniqueName: \"kubernetes.io/secret/0e99cfe5-3268-401d-b985-7634e2bac80b-tls-certs\") pod \"0e99cfe5-3268-401d-b985-7634e2bac80b\" (UID: \"0e99cfe5-3268-401d-b985-7634e2bac80b\") "
Oct 30 23:04:44.693363 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.692075    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e99cfe5-3268-401d-b985-7634e2bac80b-kube-api-access-7ftz5\" (UniqueName: \"kubernetes.io/projected/0e99cfe5-3268-401d-b985-7634e2bac80b-kube-api-access-7ftz5\") pod \"0e99cfe5-3268-401d-b985-7634e2bac80b\" (UID: \"0e99cfe5-3268-401d-b985-7634e2bac80b\") "
Oct 30 23:04:44.693363 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.692108    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-model-cache" pod "0e99cfe5-3268-401d-b985-7634e2bac80b" (UID: "0e99cfe5-3268-401d-b985-7634e2bac80b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:44.693363 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.692144    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-home" pod "5d18c6ee-2f7b-4a85-9065-6c925004973d" (UID: "5d18c6ee-2f7b-4a85-9065-6c925004973d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:44.693363 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.692462    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:44.693363 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.692484    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:44.693363 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.692496    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:44.693363 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.692623    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-tmp-dir" pod "5d18c6ee-2f7b-4a85-9065-6c925004973d" (UID: "5d18c6ee-2f7b-4a85-9065-6c925004973d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:44.695792 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.695502    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e99cfe5-3268-401d-b985-7634e2bac80b-kube-api-access-7ftz5" pod "0e99cfe5-3268-401d-b985-7634e2bac80b" (UID: "0e99cfe5-3268-401d-b985-7634e2bac80b"). InnerVolumeSpecName "kube-api-access-7ftz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:44.695792 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.695511    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-tmp-dir" pod "0e99cfe5-3268-401d-b985-7634e2bac80b" (UID: "0e99cfe5-3268-401d-b985-7634e2bac80b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:44.696056 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.696025    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-model-cache" pod "5d18c6ee-2f7b-4a85-9065-6c925004973d" (UID: "5d18c6ee-2f7b-4a85-9065-6c925004973d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:44.696453 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.696419    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d18c6ee-2f7b-4a85-9065-6c925004973d-kube-api-access-5lc5g" pod "5d18c6ee-2f7b-4a85-9065-6c925004973d" (UID: "5d18c6ee-2f7b-4a85-9065-6c925004973d"). InnerVolumeSpecName "kube-api-access-5lc5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:44.699449 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.699419    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-dshm" pod "5d18c6ee-2f7b-4a85-9065-6c925004973d" (UID: "5d18c6ee-2f7b-4a85-9065-6c925004973d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:44.699629 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.699607    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d18c6ee-2f7b-4a85-9065-6c925004973d-tls-certs" pod "5d18c6ee-2f7b-4a85-9065-6c925004973d" (UID: "5d18c6ee-2f7b-4a85-9065-6c925004973d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:44.700103 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.700076    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e99cfe5-3268-401d-b985-7634e2bac80b-tls-certs" pod "0e99cfe5-3268-401d-b985-7634e2bac80b" (UID: "0e99cfe5-3268-401d-b985-7634e2bac80b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:44.700830 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.700808    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-dshm" pod "0e99cfe5-3268-401d-b985-7634e2bac80b" (UID: "0e99cfe5-3268-401d-b985-7634e2bac80b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:44.776044 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.776008    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51bd7538-efdc-40b3-b43c-35e7714a1014" path="/var/lib/kubelet/pods/51bd7538-efdc-40b3-b43c-35e7714a1014/volumes"
Oct 30 23:04:44.776477 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.776459    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d033fb5-f7fb-4480-9cf6-57effbff1402" path="/var/lib/kubelet/pods/6d033fb5-f7fb-4480-9cf6-57effbff1402/volumes"
Oct 30 23:04:44.793826 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.793720    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e99cfe5-3268-401d-b985-7634e2bac80b-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:44.793826 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.793753    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ftz5\" (UniqueName: \"kubernetes.io/projected/0e99cfe5-3268-401d-b985-7634e2bac80b-kube-api-access-7ftz5\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:44.793826 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.793768    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d18c6ee-2f7b-4a85-9065-6c925004973d-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:44.793826 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.793783    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:44.793826 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.793795    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:44.793826 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.793808    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:44.793826 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.793820    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e99cfe5-3268-401d-b985-7634e2bac80b-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:44.794078 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.793834    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lc5g\" (UniqueName: \"kubernetes.io/projected/5d18c6ee-2f7b-4a85-9065-6c925004973d-kube-api-access-5lc5g\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:44.794078 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.793847    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d18c6ee-2f7b-4a85-9065-6c925004973d-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:44.908535 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.908438    4889 generic.go:356] "Generic (PLEG): container finished" podID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerID="d67e46092f17279a11459e9f4ef2bfffd7e07d98886da23307f1410590683891" exitCode=0
Oct 30 23:04:44.908535 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.908499    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"
Oct 30 23:04:44.908535 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.908500    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" event={"ID":"5d18c6ee-2f7b-4a85-9065-6c925004973d","Type":"ContainerDied","Data":"d67e46092f17279a11459e9f4ef2bfffd7e07d98886da23307f1410590683891"}
Oct 30 23:04:44.908839 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.908545    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46" event={"ID":"5d18c6ee-2f7b-4a85-9065-6c925004973d","Type":"ContainerDied","Data":"1d0584ecfe1c12dfdddb183bd9c8f0bd436371690081b58e092dd7fd941dd30b"}
Oct 30 23:04:44.908839 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.908571    4889 scope.go:122] "RemoveContainer" containerID="d67e46092f17279a11459e9f4ef2bfffd7e07d98886da23307f1410590683891"
Oct 30 23:04:44.910958 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.910926    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c10141-5pjp6" event={"ID":"d893ba75-1d6a-404b-be7d-8b78ca3e6c62","Type":"ContainerStarted","Data":"027bce2ba27c46152de8cbda56f7472ce1f6986de2994af09c70284933c1ac7e"}
Oct 30 23:04:44.911154 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.911135    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c10141-5pjp6" event={"ID":"d893ba75-1d6a-404b-be7d-8b78ca3e6c62","Type":"ContainerStarted","Data":"712cf55d51303cea8bf918735389c6b8895ea6879f1b0e9a5e56b30b5429ea14"}
Oct 30 23:04:44.912575 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.912550    4889 generic.go:356] "Generic (PLEG): container finished" podID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerID="ea23841308d9459ab2e480beee95231491f95a06b2621d29dd2489c61ccca6c2" exitCode=0
Oct 30 23:04:44.912680 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.912582    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"
Oct 30 23:04:44.912680 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.912639    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" event={"ID":"0e99cfe5-3268-401d-b985-7634e2bac80b","Type":"ContainerDied","Data":"ea23841308d9459ab2e480beee95231491f95a06b2621d29dd2489c61ccca6c2"}
Oct 30 23:04:44.912783 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.912680    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4" event={"ID":"0e99cfe5-3268-401d-b985-7634e2bac80b","Type":"ContainerDied","Data":"a5c5615a171f68e7bae36c2aef3689bbe69993dc1da669a29b22e6dcd65adb22"}
Oct 30 23:04:44.920283 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.920257    4889 scope.go:122] "RemoveContainer" containerID="d67e46092f17279a11459e9f4ef2bfffd7e07d98886da23307f1410590683891"
Oct 30 23:04:44.921055 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:04:44.921034    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d67e46092f17279a11459e9f4ef2bfffd7e07d98886da23307f1410590683891\": container with ID starting with d67e46092f17279a11459e9f4ef2bfffd7e07d98886da23307f1410590683891 not found: ID does not exist" containerID="d67e46092f17279a11459e9f4ef2bfffd7e07d98886da23307f1410590683891"
Oct 30 23:04:44.921127 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.921064    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67e46092f17279a11459e9f4ef2bfffd7e07d98886da23307f1410590683891"} err="failed to get container status \"d67e46092f17279a11459e9f4ef2bfffd7e07d98886da23307f1410590683891\": rpc error: code = NotFound desc = could not find container \"d67e46092f17279a11459e9f4ef2bfffd7e07d98886da23307f1410590683891\": container with ID starting with d67e46092f17279a11459e9f4ef2bfffd7e07d98886da23307f1410590683891 not found: ID does not exist"
Oct 30 23:04:44.921127 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.921082    4889 scope.go:122] "RemoveContainer" containerID="ea23841308d9459ab2e480beee95231491f95a06b2621d29dd2489c61ccca6c2"
Oct 30 23:04:44.930096 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.930075    4889 scope.go:122] "RemoveContainer" containerID="ea23841308d9459ab2e480beee95231491f95a06b2621d29dd2489c61ccca6c2"
Oct 30 23:04:44.930441 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:04:44.930423    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea23841308d9459ab2e480beee95231491f95a06b2621d29dd2489c61ccca6c2\": container with ID starting with ea23841308d9459ab2e480beee95231491f95a06b2621d29dd2489c61ccca6c2 not found: ID does not exist" containerID="ea23841308d9459ab2e480beee95231491f95a06b2621d29dd2489c61ccca6c2"
Oct 30 23:04:44.930519 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.930450    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea23841308d9459ab2e480beee95231491f95a06b2621d29dd2489c61ccca6c2"} err="failed to get container status \"ea23841308d9459ab2e480beee95231491f95a06b2621d29dd2489c61ccca6c2\": rpc error: code = NotFound desc = could not find container \"ea23841308d9459ab2e480beee95231491f95a06b2621d29dd2489c61ccca6c2\": container with ID starting with ea23841308d9459ab2e480beee95231491f95a06b2621d29dd2489c61ccca6c2 not found: ID does not exist"
Oct 30 23:04:44.967989 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.967929    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c10141-5pjp6" podStartSLOduration=1.967915317 podStartE2EDuration="1.967915317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:04:44.966118328 +0000 UTC m=+3470.322484642" watchObservedRunningTime="2026-10-30 23:04:44.967915317 +0000 UTC m=+3470.324281632"
Oct 30 23:04:44.988132 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:44.988106    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq"]
Oct 30 23:04:45.008921 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:45.008889    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"]
Oct 30 23:04:45.014255 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:45.014142    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-2b1e88/distinct-2-w1-2b1e88-kserve-5c75c467d9-8ssn4"]
Oct 30 23:04:45.049322 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:45.049289    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"]
Oct 30 23:04:45.053187 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:45.053161    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-2b1e88/premium-w1-2b1e88-kserve-5867fcd7-scd46"]
Oct 30 23:04:45.887356 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:45.887317    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main" containerID="cri-o://ed7044681b37bb733a06c65453d32e863d61664739335c4238e2da725df14571" gracePeriod=45
Oct 30 23:04:45.899190 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:45.899149    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" containerID="cri-o://8e3568151d99459f8b720d04282c5af74325a9f235b6ca40dfd45f0fdbb66212" gracePeriod=45
Oct 30 23:04:45.923319 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:45.923272    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq" event={"ID":"4c8f8098-b597-45ce-ac4b-60aa0910e60a","Type":"ContainerStarted","Data":"678617f0ad52ecb5aa16943d80d253236ba29e2b6e90c27ab57730a39782fd38"}
Oct 30 23:04:45.923489 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:45.923345    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq" event={"ID":"4c8f8098-b597-45ce-ac4b-60aa0910e60a","Type":"ContainerStarted","Data":"71249150003f6f2df541c0f56d67cbc23969c14846af6008fe15b9b823f85114"}
Oct 30 23:04:45.925903 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:45.925871    4889 generic.go:356] "Generic (PLEG): container finished" podID="d893ba75-1d6a-404b-be7d-8b78ca3e6c62" containerID="027bce2ba27c46152de8cbda56f7472ce1f6986de2994af09c70284933c1ac7e" exitCode=0
Oct 30 23:04:45.926030 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:45.925917    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c10141-5pjp6" event={"ID":"d893ba75-1d6a-404b-be7d-8b78ca3e6c62","Type":"ContainerDied","Data":"027bce2ba27c46152de8cbda56f7472ce1f6986de2994af09c70284933c1ac7e"}
Oct 30 23:04:45.938101 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:45.938068    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:45.979018 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:45.978960    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq" podStartSLOduration=1.978944442 podStartE2EDuration="1.978944442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:04:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:04:45.963408811 +0000 UTC m=+3471.319775126" watchObservedRunningTime="2026-10-30 23:04:45.978944442 +0000 UTC m=+3471.335310756"
Oct 30 23:04:46.183776 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.183750    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 23:04:46.187096 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.187074    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 23:04:46.309895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.309861    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-dshm\") pod \"6af5c6ee-e62b-4095-924b-4d4440b91a26\" (UID: \"6af5c6ee-e62b-4095-924b-4d4440b91a26\") "
Oct 30 23:04:46.310045 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.310002    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-home\" (UniqueName: \"kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-home\") pod \"6af5c6ee-e62b-4095-924b-4d4440b91a26\" (UID: \"6af5c6ee-e62b-4095-924b-4d4440b91a26\") "
Oct 30 23:04:46.310110 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.310045    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-tmp-dir\") pod \"6af5c6ee-e62b-4095-924b-4d4440b91a26\" (UID: \"6af5c6ee-e62b-4095-924b-4d4440b91a26\") "
Oct 30 23:04:46.310110 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.310077    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/009fde76-6593-492a-8c91-b566aa3f327a-tls-certs\" (UniqueName: \"kubernetes.io/secret/009fde76-6593-492a-8c91-b566aa3f327a-tls-certs\") pod \"009fde76-6593-492a-8c91-b566aa3f327a\" (UID: \"009fde76-6593-492a-8c91-b566aa3f327a\") "
Oct 30 23:04:46.310239 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.310111    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-model-cache\") pod \"009fde76-6593-492a-8c91-b566aa3f327a\" (UID: \"009fde76-6593-492a-8c91-b566aa3f327a\") "
Oct 30 23:04:46.310239 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.310144    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/009fde76-6593-492a-8c91-b566aa3f327a-kube-api-access-trh79\" (UniqueName: \"kubernetes.io/projected/009fde76-6593-492a-8c91-b566aa3f327a-kube-api-access-trh79\") pod \"009fde76-6593-492a-8c91-b566aa3f327a\" (UID: \"009fde76-6593-492a-8c91-b566aa3f327a\") "
Oct 30 23:04:46.310239 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.310169    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-home\" (UniqueName: \"kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-home\") pod \"009fde76-6593-492a-8c91-b566aa3f327a\" (UID: \"009fde76-6593-492a-8c91-b566aa3f327a\") "
Oct 30 23:04:46.310239 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.310196    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-tmp-dir\") pod \"009fde76-6593-492a-8c91-b566aa3f327a\" (UID: \"009fde76-6593-492a-8c91-b566aa3f327a\") "
Oct 30 23:04:46.310476 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.310255    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-dshm\") pod \"009fde76-6593-492a-8c91-b566aa3f327a\" (UID: \"009fde76-6593-492a-8c91-b566aa3f327a\") "
Oct 30 23:04:46.310476 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.310313    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-model-cache\") pod \"6af5c6ee-e62b-4095-924b-4d4440b91a26\" (UID: \"6af5c6ee-e62b-4095-924b-4d4440b91a26\") "
Oct 30 23:04:46.310476 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.310354    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6af5c6ee-e62b-4095-924b-4d4440b91a26-kube-api-access-jfnpq\" (UniqueName: \"kubernetes.io/projected/6af5c6ee-e62b-4095-924b-4d4440b91a26-kube-api-access-jfnpq\") pod \"6af5c6ee-e62b-4095-924b-4d4440b91a26\" (UID: \"6af5c6ee-e62b-4095-924b-4d4440b91a26\") "
Oct 30 23:04:46.310476 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.310421    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6af5c6ee-e62b-4095-924b-4d4440b91a26-tls-certs\" (UniqueName: \"kubernetes.io/secret/6af5c6ee-e62b-4095-924b-4d4440b91a26-tls-certs\") pod \"6af5c6ee-e62b-4095-924b-4d4440b91a26\" (UID: \"6af5c6ee-e62b-4095-924b-4d4440b91a26\") "
Oct 30 23:04:46.310476 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.310450    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-model-cache" pod "009fde76-6593-492a-8c91-b566aa3f327a" (UID: "009fde76-6593-492a-8c91-b566aa3f327a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:46.310753 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.310469    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-home" pod "6af5c6ee-e62b-4095-924b-4d4440b91a26" (UID: "6af5c6ee-e62b-4095-924b-4d4440b91a26"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:46.310753 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.310562    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-home" pod "009fde76-6593-492a-8c91-b566aa3f327a" (UID: "009fde76-6593-492a-8c91-b566aa3f327a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:46.311690 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.310879    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:46.311690 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.310960    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:46.311690 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.311028    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:46.311690 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.311331    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-model-cache" pod "6af5c6ee-e62b-4095-924b-4d4440b91a26" (UID: "6af5c6ee-e62b-4095-924b-4d4440b91a26"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:46.311690 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.311391    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-tmp-dir" pod "6af5c6ee-e62b-4095-924b-4d4440b91a26" (UID: "6af5c6ee-e62b-4095-924b-4d4440b91a26"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:46.311690 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.311496    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-tmp-dir" pod "009fde76-6593-492a-8c91-b566aa3f327a" (UID: "009fde76-6593-492a-8c91-b566aa3f327a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:46.313280 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.313244    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-dshm" pod "6af5c6ee-e62b-4095-924b-4d4440b91a26" (UID: "6af5c6ee-e62b-4095-924b-4d4440b91a26"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:46.313637 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.313608    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/009fde76-6593-492a-8c91-b566aa3f327a-kube-api-access-trh79" pod "009fde76-6593-492a-8c91-b566aa3f327a" (UID: "009fde76-6593-492a-8c91-b566aa3f327a"). InnerVolumeSpecName "kube-api-access-trh79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:46.313637 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.313621    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/009fde76-6593-492a-8c91-b566aa3f327a-tls-certs" pod "009fde76-6593-492a-8c91-b566aa3f327a" (UID: "009fde76-6593-492a-8c91-b566aa3f327a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:46.313827 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.313622    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6af5c6ee-e62b-4095-924b-4d4440b91a26-tls-certs" pod "6af5c6ee-e62b-4095-924b-4d4440b91a26" (UID: "6af5c6ee-e62b-4095-924b-4d4440b91a26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:46.314586 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.314554    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af5c6ee-e62b-4095-924b-4d4440b91a26-kube-api-access-jfnpq" pod "6af5c6ee-e62b-4095-924b-4d4440b91a26" (UID: "6af5c6ee-e62b-4095-924b-4d4440b91a26"). InnerVolumeSpecName "kube-api-access-jfnpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:46.314743 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.314718    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-dshm" pod "009fde76-6593-492a-8c91-b566aa3f327a" (UID: "009fde76-6593-492a-8c91-b566aa3f327a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:46.412524 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.412437    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:46.412524 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.412468    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/009fde76-6593-492a-8c91-b566aa3f327a-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:46.412524 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.412478    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:46.412524 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.412488    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfnpq\" (UniqueName: \"kubernetes.io/projected/6af5c6ee-e62b-4095-924b-4d4440b91a26-kube-api-access-jfnpq\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:46.412524 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.412497    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6af5c6ee-e62b-4095-924b-4d4440b91a26-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:46.412524 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.412505    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:46.412524 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.412513    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6af5c6ee-e62b-4095-924b-4d4440b91a26-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:46.412524 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.412521    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/009fde76-6593-492a-8c91-b566aa3f327a-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:46.412958 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.412530    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trh79\" (UniqueName: \"kubernetes.io/projected/009fde76-6593-492a-8c91-b566aa3f327a-kube-api-access-trh79\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:46.773024 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.772986    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" path="/var/lib/kubelet/pods/0e99cfe5-3268-401d-b985-7634e2bac80b/volumes"
Oct 30 23:04:46.773553 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.773532    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" path="/var/lib/kubelet/pods/5d18c6ee-2f7b-4a85-9065-6c925004973d/volumes"
Oct 30 23:04:46.934886 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.934847    4889 generic.go:356] "Generic (PLEG): container finished" podID="009fde76-6593-492a-8c91-b566aa3f327a" containerID="ed7044681b37bb733a06c65453d32e863d61664739335c4238e2da725df14571" exitCode=0
Oct 30 23:04:46.935387 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.934900    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"
Oct 30 23:04:46.935387 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.934933    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" event={"ID":"009fde76-6593-492a-8c91-b566aa3f327a","Type":"ContainerDied","Data":"ed7044681b37bb733a06c65453d32e863d61664739335c4238e2da725df14571"}
Oct 30 23:04:46.935387 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.934969    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc" event={"ID":"009fde76-6593-492a-8c91-b566aa3f327a","Type":"ContainerDied","Data":"e544b1da1ff67e90e3efdbd91b2e4818704690c6a7f731c70f0321eab63176ca"}
Oct 30 23:04:46.935387 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.934992    4889 scope.go:122] "RemoveContainer" containerID="ed7044681b37bb733a06c65453d32e863d61664739335c4238e2da725df14571"
Oct 30 23:04:46.936442 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.936415    4889 generic.go:356] "Generic (PLEG): container finished" podID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerID="8e3568151d99459f8b720d04282c5af74325a9f235b6ca40dfd45f0fdbb66212" exitCode=0
Oct 30 23:04:46.936559 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.936466    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"
Oct 30 23:04:46.936559 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.936500    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" event={"ID":"6af5c6ee-e62b-4095-924b-4d4440b91a26","Type":"ContainerDied","Data":"8e3568151d99459f8b720d04282c5af74325a9f235b6ca40dfd45f0fdbb66212"}
Oct 30 23:04:46.936559 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.936530    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" event={"ID":"6af5c6ee-e62b-4095-924b-4d4440b91a26","Type":"ContainerDied","Data":"0ae6bcf98c17623064d288d006d762dbba06c751202c1cf4f2c8a0f78919e936"}
Oct 30 23:04:46.938422 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.938396    4889 generic.go:356] "Generic (PLEG): container finished" podID="4c8f8098-b597-45ce-ac4b-60aa0910e60a" containerID="678617f0ad52ecb5aa16943d80d253236ba29e2b6e90c27ab57730a39782fd38" exitCode=0
Oct 30 23:04:46.938540 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.938453    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq" event={"ID":"4c8f8098-b597-45ce-ac4b-60aa0910e60a","Type":"ContainerDied","Data":"678617f0ad52ecb5aa16943d80d253236ba29e2b6e90c27ab57730a39782fd38"}
Oct 30 23:04:46.944990 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.944971    4889 scope.go:122] "RemoveContainer" containerID="ed7044681b37bb733a06c65453d32e863d61664739335c4238e2da725df14571"
Oct 30 23:04:46.945329 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:04:46.945309    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed7044681b37bb733a06c65453d32e863d61664739335c4238e2da725df14571\": container with ID starting with ed7044681b37bb733a06c65453d32e863d61664739335c4238e2da725df14571 not found: ID does not exist" containerID="ed7044681b37bb733a06c65453d32e863d61664739335c4238e2da725df14571"
Oct 30 23:04:46.945426 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.945336    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed7044681b37bb733a06c65453d32e863d61664739335c4238e2da725df14571"} err="failed to get container status \"ed7044681b37bb733a06c65453d32e863d61664739335c4238e2da725df14571\": rpc error: code = NotFound desc = could not find container \"ed7044681b37bb733a06c65453d32e863d61664739335c4238e2da725df14571\": container with ID starting with ed7044681b37bb733a06c65453d32e863d61664739335c4238e2da725df14571 not found: ID does not exist"
Oct 30 23:04:46.945426 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.945357    4889 scope.go:122] "RemoveContainer" containerID="8e3568151d99459f8b720d04282c5af74325a9f235b6ca40dfd45f0fdbb66212"
Oct 30 23:04:46.956156 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.956118    4889 scope.go:122] "RemoveContainer" containerID="8e3568151d99459f8b720d04282c5af74325a9f235b6ca40dfd45f0fdbb66212"
Oct 30 23:04:46.956602 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:04:46.956486    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e3568151d99459f8b720d04282c5af74325a9f235b6ca40dfd45f0fdbb66212\": container with ID starting with 8e3568151d99459f8b720d04282c5af74325a9f235b6ca40dfd45f0fdbb66212 not found: ID does not exist" containerID="8e3568151d99459f8b720d04282c5af74325a9f235b6ca40dfd45f0fdbb66212"
Oct 30 23:04:46.956602 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:46.956512    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e3568151d99459f8b720d04282c5af74325a9f235b6ca40dfd45f0fdbb66212"} err="failed to get container status \"8e3568151d99459f8b720d04282c5af74325a9f235b6ca40dfd45f0fdbb66212\": rpc error: code = NotFound desc = could not find container \"8e3568151d99459f8b720d04282c5af74325a9f235b6ca40dfd45f0fdbb66212\": container with ID starting with 8e3568151d99459f8b720d04282c5af74325a9f235b6ca40dfd45f0fdbb66212 not found: ID does not exist"
Oct 30 23:04:47.035687 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:47.035583    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"]
Oct 30 23:04:47.038767 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:47.038739    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x"]
Oct 30 23:04:47.076863 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:47.076823    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"]
Oct 30 23:04:47.084284 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:47.084251    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-09d0d5/distinct-w0-09d0d5-kserve-5db7449545-qlgvc"]
Oct 30 23:04:47.091554 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:47.091522    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c10141-5pjp6"
Oct 30 23:04:47.154911 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:47.154865    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/premium-w0-09d0d5-kserve-8b7f66bb6-w8l5x" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:04:47.221170 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:47.221135    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d893ba75-1d6a-404b-be7d-8b78ca3e6c62-kube-api-access-2bsz2\" (UniqueName: \"kubernetes.io/projected/d893ba75-1d6a-404b-be7d-8b78ca3e6c62-kube-api-access-2bsz2\") pod \"d893ba75-1d6a-404b-be7d-8b78ca3e6c62\" (UID: \"d893ba75-1d6a-404b-be7d-8b78ca3e6c62\") "
Oct 30 23:04:47.221363 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:47.221270    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d893ba75-1d6a-404b-be7d-8b78ca3e6c62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d893ba75-1d6a-404b-be7d-8b78ca3e6c62-maas-api-xks-ca\") pod \"d893ba75-1d6a-404b-be7d-8b78ca3e6c62\" (UID: \"d893ba75-1d6a-404b-be7d-8b78ca3e6c62\") "
Oct 30 23:04:47.221363 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:47.221302    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d893ba75-1d6a-404b-be7d-8b78ca3e6c62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d893ba75-1d6a-404b-be7d-8b78ca3e6c62-maas-api-openshift-ca\") pod \"d893ba75-1d6a-404b-be7d-8b78ca3e6c62\" (UID: \"d893ba75-1d6a-404b-be7d-8b78ca3e6c62\") "
Oct 30 23:04:47.221720 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:47.221697    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d893ba75-1d6a-404b-be7d-8b78ca3e6c62-maas-api-openshift-ca" pod "d893ba75-1d6a-404b-be7d-8b78ca3e6c62" (UID: "d893ba75-1d6a-404b-be7d-8b78ca3e6c62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:47.223653 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:47.223623    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d893ba75-1d6a-404b-be7d-8b78ca3e6c62-kube-api-access-2bsz2" pod "d893ba75-1d6a-404b-be7d-8b78ca3e6c62" (UID: "d893ba75-1d6a-404b-be7d-8b78ca3e6c62"). InnerVolumeSpecName "kube-api-access-2bsz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:47.223653 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:47.223636    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d893ba75-1d6a-404b-be7d-8b78ca3e6c62-maas-api-xks-ca" pod "d893ba75-1d6a-404b-be7d-8b78ca3e6c62" (UID: "d893ba75-1d6a-404b-be7d-8b78ca3e6c62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:47.322693 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:47.322606    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bsz2\" (UniqueName: \"kubernetes.io/projected/d893ba75-1d6a-404b-be7d-8b78ca3e6c62-kube-api-access-2bsz2\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:47.322693 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:47.322636    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d893ba75-1d6a-404b-be7d-8b78ca3e6c62-maas-api-xks-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:47.322693 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:47.322645    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d893ba75-1d6a-404b-be7d-8b78ca3e6c62-maas-api-openshift-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:47.726553 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:47.726454    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:47.948879 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:47.948849    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c10141-5pjp6"
Oct 30 23:04:47.949327 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:47.948874    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c10141-5pjp6" event={"ID":"d893ba75-1d6a-404b-be7d-8b78ca3e6c62","Type":"ContainerDied","Data":"712cf55d51303cea8bf918735389c6b8895ea6879f1b0e9a5e56b30b5429ea14"}
Oct 30 23:04:47.949327 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:47.948924    4889 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="712cf55d51303cea8bf918735389c6b8895ea6879f1b0e9a5e56b30b5429ea14"
Oct 30 23:04:48.091660 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:48.091628    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq"
Oct 30 23:04:48.230520 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:48.230485    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c8f8098-b597-45ce-ac4b-60aa0910e60a-kube-api-access-r5rvn\" (UniqueName: \"kubernetes.io/projected/4c8f8098-b597-45ce-ac4b-60aa0910e60a-kube-api-access-r5rvn\") pod \"4c8f8098-b597-45ce-ac4b-60aa0910e60a\" (UID: \"4c8f8098-b597-45ce-ac4b-60aa0910e60a\") "
Oct 30 23:04:48.230683 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:48.230571    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c8f8098-b597-45ce-ac4b-60aa0910e60a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8f8098-b597-45ce-ac4b-60aa0910e60a-maas-api-openshift-ca\") pod \"4c8f8098-b597-45ce-ac4b-60aa0910e60a\" (UID: \"4c8f8098-b597-45ce-ac4b-60aa0910e60a\") "
Oct 30 23:04:48.230683 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:48.230629    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c8f8098-b597-45ce-ac4b-60aa0910e60a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8f8098-b597-45ce-ac4b-60aa0910e60a-maas-api-xks-ca\") pod \"4c8f8098-b597-45ce-ac4b-60aa0910e60a\" (UID: \"4c8f8098-b597-45ce-ac4b-60aa0910e60a\") "
Oct 30 23:04:48.230988 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:48.230954    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c8f8098-b597-45ce-ac4b-60aa0910e60a-maas-api-openshift-ca" pod "4c8f8098-b597-45ce-ac4b-60aa0910e60a" (UID: "4c8f8098-b597-45ce-ac4b-60aa0910e60a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:48.232976 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:48.232946    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c8f8098-b597-45ce-ac4b-60aa0910e60a-maas-api-xks-ca" pod "4c8f8098-b597-45ce-ac4b-60aa0910e60a" (UID: "4c8f8098-b597-45ce-ac4b-60aa0910e60a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:48.233089 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:48.233063    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c8f8098-b597-45ce-ac4b-60aa0910e60a-kube-api-access-r5rvn" pod "4c8f8098-b597-45ce-ac4b-60aa0910e60a" (UID: "4c8f8098-b597-45ce-ac4b-60aa0910e60a"). InnerVolumeSpecName "kube-api-access-r5rvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:48.331600 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:48.331508    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8f8098-b597-45ce-ac4b-60aa0910e60a-maas-api-xks-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:48.331600 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:48.331548    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5rvn\" (UniqueName: \"kubernetes.io/projected/4c8f8098-b597-45ce-ac4b-60aa0910e60a-kube-api-access-r5rvn\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:48.331600 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:48.331564    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8f8098-b597-45ce-ac4b-60aa0910e60a-maas-api-openshift-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:48.754108 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:48.754069    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:48.773032 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:48.772996    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="009fde76-6593-492a-8c91-b566aa3f327a" path="/var/lib/kubelet/pods/009fde76-6593-492a-8c91-b566aa3f327a/volumes"
Oct 30 23:04:48.773562 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:48.773540    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" path="/var/lib/kubelet/pods/6af5c6ee-e62b-4095-924b-4d4440b91a26/volumes"
Oct 30 23:04:48.920780 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:48.920744    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:48.956432 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:48.956400    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq" event={"ID":"4c8f8098-b597-45ce-ac4b-60aa0910e60a","Type":"ContainerDied","Data":"71249150003f6f2df541c0f56d67cbc23969c14846af6008fe15b9b823f85114"}
Oct 30 23:04:48.956822 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:48.956435    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq"
Oct 30 23:04:48.956822 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:48.956447    4889 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71249150003f6f2df541c0f56d67cbc23969c14846af6008fe15b9b823f85114"
Oct 30 23:04:49.003261 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:49.003206    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"]
Oct 30 23:04:49.003532 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:49.003503    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp" podUID="b69c09df-8ced-418f-92f4-6c9ead312931" containerName="istio-proxy" containerID="cri-o://67df2fd3ffc77e25f94f95889dcc7ef39e0488f2859f3f0889073ff02dbd0fd2" gracePeriod=30
Oct 30 23:04:49.089446 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:49.088111    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:49.418550 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:49.418452    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:51.338606 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:51.338571    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:51.836626 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:51.836589    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:53.168531 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:53.168492    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:54.264992 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.264966    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 23:04:54.388899 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.388813    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b69c09df-8ced-418f-92f4-6c9ead312931-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b69c09df-8ced-418f-92f4-6c9ead312931-istio-podinfo\") pod \"b69c09df-8ced-418f-92f4-6c9ead312931\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") "
Oct 30 23:04:54.388899 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.388875    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b69c09df-8ced-418f-92f4-6c9ead312931-istio-token\" (UniqueName: \"kubernetes.io/projected/b69c09df-8ced-418f-92f4-6c9ead312931-istio-token\") pod \"b69c09df-8ced-418f-92f4-6c9ead312931\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") "
Oct 30 23:04:54.389118 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.388934    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-workload-certs\") pod \"b69c09df-8ced-418f-92f4-6c9ead312931\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") "
Oct 30 23:04:54.389118 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.388963    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-credential-socket\") pod \"b69c09df-8ced-418f-92f4-6c9ead312931\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") "
Oct 30 23:04:54.389118 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.389001    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b69c09df-8ced-418f-92f4-6c9ead312931-kube-api-access-pq5rr\" (UniqueName: \"kubernetes.io/projected/b69c09df-8ced-418f-92f4-6c9ead312931-kube-api-access-pq5rr\") pod \"b69c09df-8ced-418f-92f4-6c9ead312931\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") "
Oct 30 23:04:54.389118 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.389058    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b69c09df-8ced-418f-92f4-6c9ead312931-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b69c09df-8ced-418f-92f4-6c9ead312931-istiod-ca-cert\") pod \"b69c09df-8ced-418f-92f4-6c9ead312931\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") "
Oct 30 23:04:54.389118 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.389098    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-istio-envoy\") pod \"b69c09df-8ced-418f-92f4-6c9ead312931\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") "
Oct 30 23:04:54.389437 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.389138    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-istio-data\") pod \"b69c09df-8ced-418f-92f4-6c9ead312931\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") "
Oct 30 23:04:54.389437 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.389192    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-workload-socket\") pod \"b69c09df-8ced-418f-92f4-6c9ead312931\" (UID: \"b69c09df-8ced-418f-92f4-6c9ead312931\") "
Oct 30 23:04:54.389437 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.389347    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-workload-certs" pod "b69c09df-8ced-418f-92f4-6c9ead312931" (UID: "b69c09df-8ced-418f-92f4-6c9ead312931"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:54.389437 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.389366    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-credential-socket" pod "b69c09df-8ced-418f-92f4-6c9ead312931" (UID: "b69c09df-8ced-418f-92f4-6c9ead312931"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:54.389644 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.389569    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-istio-data" pod "b69c09df-8ced-418f-92f4-6c9ead312931" (UID: "b69c09df-8ced-418f-92f4-6c9ead312931"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:54.389644 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.389631    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-workload-socket" pod "b69c09df-8ced-418f-92f4-6c9ead312931" (UID: "b69c09df-8ced-418f-92f4-6c9ead312931"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:54.389771 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.389644    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:54.389771 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.389648    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b69c09df-8ced-418f-92f4-6c9ead312931-istiod-ca-cert" pod "b69c09df-8ced-418f-92f4-6c9ead312931" (UID: "b69c09df-8ced-418f-92f4-6c9ead312931"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:54.389771 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.389666    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:54.391357 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.391332    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b69c09df-8ced-418f-92f4-6c9ead312931-istio-podinfo" pod "b69c09df-8ced-418f-92f4-6c9ead312931" (UID: "b69c09df-8ced-418f-92f4-6c9ead312931"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:04:54.391357 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.391341    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b69c09df-8ced-418f-92f4-6c9ead312931-istio-token" pod "b69c09df-8ced-418f-92f4-6c9ead312931" (UID: "b69c09df-8ced-418f-92f4-6c9ead312931"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:54.391616 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.391593    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b69c09df-8ced-418f-92f4-6c9ead312931-kube-api-access-pq5rr" pod "b69c09df-8ced-418f-92f4-6c9ead312931" (UID: "b69c09df-8ced-418f-92f4-6c9ead312931"). InnerVolumeSpecName "kube-api-access-pq5rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:54.391726 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.391620    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-istio-envoy" pod "b69c09df-8ced-418f-92f4-6c9ead312931" (UID: "b69c09df-8ced-418f-92f4-6c9ead312931"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:54.490985 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.490950    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b69c09df-8ced-418f-92f4-6c9ead312931-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:54.490985 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.490981    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b69c09df-8ced-418f-92f4-6c9ead312931-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:54.490985 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.490991    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq5rr\" (UniqueName: \"kubernetes.io/projected/b69c09df-8ced-418f-92f4-6c9ead312931-kube-api-access-pq5rr\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:54.491328 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.491003    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b69c09df-8ced-418f-92f4-6c9ead312931-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:54.491328 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.491011    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:54.491328 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.491023    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:54.491328 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.491035    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b69c09df-8ced-418f-92f4-6c9ead312931-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:04:54.701075 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.700981    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:54.779156 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.779125    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5"]
Oct 30 23:04:54.914734 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.914680    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:54.991708 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.991675    4889 generic.go:356] "Generic (PLEG): container finished" podID="b69c09df-8ced-418f-92f4-6c9ead312931" containerID="67df2fd3ffc77e25f94f95889dcc7ef39e0488f2859f3f0889073ff02dbd0fd2" exitCode=0
Oct 30 23:04:54.991863 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.991724    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"
Oct 30 23:04:54.991863 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.991754    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp" event={"ID":"b69c09df-8ced-418f-92f4-6c9ead312931","Type":"ContainerDied","Data":"67df2fd3ffc77e25f94f95889dcc7ef39e0488f2859f3f0889073ff02dbd0fd2"}
Oct 30 23:04:54.991863 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.991807    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp" event={"ID":"b69c09df-8ced-418f-92f4-6c9ead312931","Type":"ContainerDied","Data":"5c5b97c8ddf0022e0b7032a64640da32511064e98b96caa515e9ae5488d25440"}
Oct 30 23:04:54.991863 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:54.991828    4889 scope.go:122] "RemoveContainer" containerID="67df2fd3ffc77e25f94f95889dcc7ef39e0488f2859f3f0889073ff02dbd0fd2"
Oct 30 23:04:55.002010 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:55.001993    4889 scope.go:122] "RemoveContainer" containerID="67df2fd3ffc77e25f94f95889dcc7ef39e0488f2859f3f0889073ff02dbd0fd2"
Oct 30 23:04:55.002300 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:04:55.002281    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67df2fd3ffc77e25f94f95889dcc7ef39e0488f2859f3f0889073ff02dbd0fd2\": container with ID starting with 67df2fd3ffc77e25f94f95889dcc7ef39e0488f2859f3f0889073ff02dbd0fd2 not found: ID does not exist" containerID="67df2fd3ffc77e25f94f95889dcc7ef39e0488f2859f3f0889073ff02dbd0fd2"
Oct 30 23:04:55.002351 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:55.002310    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67df2fd3ffc77e25f94f95889dcc7ef39e0488f2859f3f0889073ff02dbd0fd2"} err="failed to get container status \"67df2fd3ffc77e25f94f95889dcc7ef39e0488f2859f3f0889073ff02dbd0fd2\": rpc error: code = NotFound desc = could not find container \"67df2fd3ffc77e25f94f95889dcc7ef39e0488f2859f3f0889073ff02dbd0fd2\": container with ID starting with 67df2fd3ffc77e25f94f95889dcc7ef39e0488f2859f3f0889073ff02dbd0fd2 not found: ID does not exist"
Oct 30 23:04:55.047726 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:55.047693    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"]
Oct 30 23:04:55.053802 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:55.053778    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k94dp"]
Oct 30 23:04:55.090582 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:55.090547    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:55.434669 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:55.434572    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:56.073594 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:56.073556    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:56.144216 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:56.144180    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq"]
Oct 30 23:04:56.157309 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:56.157277    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cdc122-bfsgq"]
Oct 30 23:04:56.262426 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:56.262384    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:56.686596 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:56.686563    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:56.773280 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:56.773248    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c8f8098-b597-45ce-ac4b-60aa0910e60a" path="/var/lib/kubelet/pods/4c8f8098-b597-45ce-ac4b-60aa0910e60a/volumes"
Oct 30 23:04:56.773699 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:56.773686    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b69c09df-8ced-418f-92f4-6c9ead312931" path="/var/lib/kubelet/pods/b69c09df-8ced-418f-92f4-6c9ead312931/volumes"
Oct 30 23:04:57.421171 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:57.421135    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:57.958777 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:57.958744    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:58.154346 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:58.154312    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:59.423801 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:59.423763    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:59.779440 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:04:59.779400    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5" podUID="e1595760-0ede-4e19-ad58-741f6f577207" containerName="payload-processing" containerID="cri-o://290e72c21f8ca404ec8317466ff1623eed8e702e7f36a523208ad6bd9e2d6398" gracePeriod=25
Oct 30 23:05:01.114698 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:01.114662    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:01.307676 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:01.307637    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:01.787669 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:01.787624    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5" podUID="e1595760-0ede-4e19-ad58-741f6f577207" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:05:02.711303 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:02.711264    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6"]
Oct 30 23:05:02.712003 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:02.711534    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6" podUID="44c69100-209c-49c3-9c84-cb2d36028de5" containerName="maas-api" containerID="cri-o://a213e973df03acd4bc29a61a6c8adbeac2350142409f42bdea7a7ad902cc44fe" gracePeriod=30
Oct 30 23:05:02.830010 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:02.829973    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:02.974165 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:02.974102    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6"
Oct 30 23:05:03.040043 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:03.039993    4889 generic.go:356] "Generic (PLEG): container finished" podID="44c69100-209c-49c3-9c84-cb2d36028de5" containerID="a213e973df03acd4bc29a61a6c8adbeac2350142409f42bdea7a7ad902cc44fe" exitCode=0
Oct 30 23:05:03.040043 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:03.040042    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6"
Oct 30 23:05:03.040311 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:03.040072    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6" event={"ID":"44c69100-209c-49c3-9c84-cb2d36028de5","Type":"ContainerDied","Data":"a213e973df03acd4bc29a61a6c8adbeac2350142409f42bdea7a7ad902cc44fe"}
Oct 30 23:05:03.040311 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:03.040140    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6" event={"ID":"44c69100-209c-49c3-9c84-cb2d36028de5","Type":"ContainerDied","Data":"9b0d183fbc345efe19d430ad224ca6c50c31a7d547770759160e02563aaf00cc"}
Oct 30 23:05:03.040311 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:03.040174    4889 scope.go:122] "RemoveContainer" containerID="a213e973df03acd4bc29a61a6c8adbeac2350142409f42bdea7a7ad902cc44fe"
Oct 30 23:05:03.050797 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:03.050767    4889 scope.go:122] "RemoveContainer" containerID="a213e973df03acd4bc29a61a6c8adbeac2350142409f42bdea7a7ad902cc44fe"
Oct 30 23:05:03.051063 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:05:03.051044    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a213e973df03acd4bc29a61a6c8adbeac2350142409f42bdea7a7ad902cc44fe\": container with ID starting with a213e973df03acd4bc29a61a6c8adbeac2350142409f42bdea7a7ad902cc44fe not found: ID does not exist" containerID="a213e973df03acd4bc29a61a6c8adbeac2350142409f42bdea7a7ad902cc44fe"
Oct 30 23:05:03.051120 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:03.051076    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a213e973df03acd4bc29a61a6c8adbeac2350142409f42bdea7a7ad902cc44fe"} err="failed to get container status \"a213e973df03acd4bc29a61a6c8adbeac2350142409f42bdea7a7ad902cc44fe\": rpc error: code = NotFound desc = could not find container \"a213e973df03acd4bc29a61a6c8adbeac2350142409f42bdea7a7ad902cc44fe\": container with ID starting with a213e973df03acd4bc29a61a6c8adbeac2350142409f42bdea7a7ad902cc44fe not found: ID does not exist"
Oct 30 23:05:03.077310 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:03.077276    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44c69100-209c-49c3-9c84-cb2d36028de5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/44c69100-209c-49c3-9c84-cb2d36028de5-metrics-certs\") pod \"44c69100-209c-49c3-9c84-cb2d36028de5\" (UID: \"44c69100-209c-49c3-9c84-cb2d36028de5\") "
Oct 30 23:05:03.077471 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:03.077347    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44c69100-209c-49c3-9c84-cb2d36028de5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44c69100-209c-49c3-9c84-cb2d36028de5-openshift-service-ca\") pod \"44c69100-209c-49c3-9c84-cb2d36028de5\" (UID: \"44c69100-209c-49c3-9c84-cb2d36028de5\") "
Oct 30 23:05:03.077471 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:03.077366    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44c69100-209c-49c3-9c84-cb2d36028de5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44c69100-209c-49c3-9c84-cb2d36028de5-maas-api-tls\") pod \"44c69100-209c-49c3-9c84-cb2d36028de5\" (UID: \"44c69100-209c-49c3-9c84-cb2d36028de5\") "
Oct 30 23:05:03.077471 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:03.077400    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44c69100-209c-49c3-9c84-cb2d36028de5-kube-api-access-n46hf\" (UniqueName: \"kubernetes.io/projected/44c69100-209c-49c3-9c84-cb2d36028de5-kube-api-access-n46hf\") pod \"44c69100-209c-49c3-9c84-cb2d36028de5\" (UID: \"44c69100-209c-49c3-9c84-cb2d36028de5\") "
Oct 30 23:05:03.077789 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:03.077764    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44c69100-209c-49c3-9c84-cb2d36028de5-openshift-service-ca" pod "44c69100-209c-49c3-9c84-cb2d36028de5" (UID: "44c69100-209c-49c3-9c84-cb2d36028de5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:05:03.079632 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:03.079601    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44c69100-209c-49c3-9c84-cb2d36028de5-metrics-certs" pod "44c69100-209c-49c3-9c84-cb2d36028de5" (UID: "44c69100-209c-49c3-9c84-cb2d36028de5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:05:03.079724 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:03.079669    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44c69100-209c-49c3-9c84-cb2d36028de5-maas-api-tls" pod "44c69100-209c-49c3-9c84-cb2d36028de5" (UID: "44c69100-209c-49c3-9c84-cb2d36028de5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:05:03.079724 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:03.079672    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c69100-209c-49c3-9c84-cb2d36028de5-kube-api-access-n46hf" pod "44c69100-209c-49c3-9c84-cb2d36028de5" (UID: "44c69100-209c-49c3-9c84-cb2d36028de5"). InnerVolumeSpecName "kube-api-access-n46hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:03.178796 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:03.178760    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44c69100-209c-49c3-9c84-cb2d36028de5-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:05:03.178796 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:03.178794    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44c69100-209c-49c3-9c84-cb2d36028de5-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:05:03.179006 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:03.178805    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44c69100-209c-49c3-9c84-cb2d36028de5-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:05:03.179006 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:03.178813    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n46hf\" (UniqueName: \"kubernetes.io/projected/44c69100-209c-49c3-9c84-cb2d36028de5-kube-api-access-n46hf\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:05:03.321114 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:03.321079    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:03.388469 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:03.388434    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6"]
Oct 30 23:05:03.392482 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:03.392453    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-09d0d5-5847b7b79c-fkps6"]
Oct 30 23:05:03.512015 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:03.511980    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:03.995560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:03.995511    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:04.172548 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:04.172512    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:04.480373 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:04.480259    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:04.705096 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:04.705059    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:04.772647 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:04.772609    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44c69100-209c-49c3-9c84-cb2d36028de5" path="/var/lib/kubelet/pods/44c69100-209c-49c3-9c84-cb2d36028de5/volumes"
Oct 30 23:05:04.916506 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:04.916463    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:06.182086 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:06.182040    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:06.354864 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:06.354823    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:08.030326 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:08.030288    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:08.430508 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:08.430391    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:08.684652 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:08.684544    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:08.862236 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:08.862190    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:09.353730 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:09.353641    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:09.912520 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:09.912476    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:10.302325 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:05:10.302286    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:05:11.788332 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:11.788295    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5" podUID="e1595760-0ede-4e19-ad58-741f6f577207" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:05:11.862092 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:11.862063    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:12.043779 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:12.043675    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:13.536204 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:13.536165    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:13.746822 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:13.746780    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:13.975801 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:13.975709    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:14.412270 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:14.412198    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:16.277109 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:16.277075    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:16.622525 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:16.622423    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:17.109810 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:17.109775    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:18.050379 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:18.050345    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:18.738580 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:18.738545    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:19.502278 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:19.502211    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:20.034129 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:20.034098    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:20.036337 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:20.036287    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5"
Oct 30 23:05:20.129344 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:20.129304    4889 generic.go:356] "Generic (PLEG): container finished" podID="e1595760-0ede-4e19-ad58-741f6f577207" containerID="290e72c21f8ca404ec8317466ff1623eed8e702e7f36a523208ad6bd9e2d6398" exitCode=0
Oct 30 23:05:20.129517 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:20.129359    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5"
Oct 30 23:05:20.129517 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:20.129390    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5" event={"ID":"e1595760-0ede-4e19-ad58-741f6f577207","Type":"ContainerDied","Data":"290e72c21f8ca404ec8317466ff1623eed8e702e7f36a523208ad6bd9e2d6398"}
Oct 30 23:05:20.129517 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:20.129449    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5" event={"ID":"e1595760-0ede-4e19-ad58-741f6f577207","Type":"ContainerDied","Data":"830d1d4db961e5de1e57f549e10082de79f65b3848dbdab774aa4e0cfcab63c3"}
Oct 30 23:05:20.129517 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:20.129474    4889 scope.go:122] "RemoveContainer" containerID="290e72c21f8ca404ec8317466ff1623eed8e702e7f36a523208ad6bd9e2d6398"
Oct 30 23:05:20.131745 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:20.131723    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1595760-0ede-4e19-ad58-741f6f577207-kube-api-access-6dcnf\" (UniqueName: \"kubernetes.io/projected/e1595760-0ede-4e19-ad58-741f6f577207-kube-api-access-6dcnf\") pod \"e1595760-0ede-4e19-ad58-741f6f577207\" (UID: \"e1595760-0ede-4e19-ad58-741f6f577207\") "
Oct 30 23:05:20.131875 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:20.131839    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1595760-0ede-4e19-ad58-741f6f577207-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1595760-0ede-4e19-ad58-741f6f577207-plugins-config-volume\") pod \"e1595760-0ede-4e19-ad58-741f6f577207\" (UID: \"e1595760-0ede-4e19-ad58-741f6f577207\") "
Oct 30 23:05:20.132199 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:20.132179    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1595760-0ede-4e19-ad58-741f6f577207-plugins-config-volume" pod "e1595760-0ede-4e19-ad58-741f6f577207" (UID: "e1595760-0ede-4e19-ad58-741f6f577207"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:05:20.134079 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:20.134053    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1595760-0ede-4e19-ad58-741f6f577207-kube-api-access-6dcnf" pod "e1595760-0ede-4e19-ad58-741f6f577207" (UID: "e1595760-0ede-4e19-ad58-741f6f577207"). InnerVolumeSpecName "kube-api-access-6dcnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:20.145776 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:20.145752    4889 scope.go:122] "RemoveContainer" containerID="290e72c21f8ca404ec8317466ff1623eed8e702e7f36a523208ad6bd9e2d6398"
Oct 30 23:05:20.146158 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:05:20.146136    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"290e72c21f8ca404ec8317466ff1623eed8e702e7f36a523208ad6bd9e2d6398\": container with ID starting with 290e72c21f8ca404ec8317466ff1623eed8e702e7f36a523208ad6bd9e2d6398 not found: ID does not exist" containerID="290e72c21f8ca404ec8317466ff1623eed8e702e7f36a523208ad6bd9e2d6398"
Oct 30 23:05:20.146273 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:20.146169    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"290e72c21f8ca404ec8317466ff1623eed8e702e7f36a523208ad6bd9e2d6398"} err="failed to get container status \"290e72c21f8ca404ec8317466ff1623eed8e702e7f36a523208ad6bd9e2d6398\": rpc error: code = NotFound desc = could not find container \"290e72c21f8ca404ec8317466ff1623eed8e702e7f36a523208ad6bd9e2d6398\": container with ID starting with 290e72c21f8ca404ec8317466ff1623eed8e702e7f36a523208ad6bd9e2d6398 not found: ID does not exist"
Oct 30 23:05:20.232910 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:20.232867    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1595760-0ede-4e19-ad58-741f6f577207-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:05:20.232910 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:20.232900    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dcnf\" (UniqueName: \"kubernetes.io/projected/e1595760-0ede-4e19-ad58-741f6f577207-kube-api-access-6dcnf\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:05:20.240251 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:20.240206    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:20.476932 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:20.476896    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5"]
Oct 30 23:05:20.480508 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:20.480481    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cdc122-5584f9d75c-fljm5"]
Oct 30 23:05:20.773531 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:20.773487    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1595760-0ede-4e19-ad58-741f6f577207" path="/var/lib/kubelet/pods/e1595760-0ede-4e19-ad58-741f6f577207/volumes"
Oct 30 23:05:20.795057 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:20.795019    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:22.170168 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:22.170135    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:24.232645 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:24.232587    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:26.246973 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:26.246930    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:27.564429 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:27.564388    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:28.332996 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:28.332956    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:29.315739 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:29.315697    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:29.495446 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:29.495409    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:29.588394 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:29.588293    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"]
Oct 30 23:05:29.588629 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:29.588598    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k" podUID="e86e2368-d03b-4634-8245-eddf9d83e16f" containerName="istio-proxy" containerID="cri-o://2251e712dfc7caa34dcbd297d00dec6b23381762077159b56f5b7a1b1651ecfe" gracePeriod=30
Oct 30 23:05:29.678537 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:29.678498    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:30.124540 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:30.124484    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:30.326961 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:30.326923    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:31.941995 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:31.941959    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:33.723494 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:33.723447    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:34.842567 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:34.842542    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 23:05:34.964941 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:34.964846    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e86e2368-d03b-4634-8245-eddf9d83e16f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e86e2368-d03b-4634-8245-eddf9d83e16f-istiod-ca-cert\") pod \"e86e2368-d03b-4634-8245-eddf9d83e16f\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") "
Oct 30 23:05:34.964941 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:34.964884    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e86e2368-d03b-4634-8245-eddf9d83e16f-kube-api-access-49l74\" (UniqueName: \"kubernetes.io/projected/e86e2368-d03b-4634-8245-eddf9d83e16f-kube-api-access-49l74\") pod \"e86e2368-d03b-4634-8245-eddf9d83e16f\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") "
Oct 30 23:05:34.964941 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:34.964911    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-podinfo\") pod \"e86e2368-d03b-4634-8245-eddf9d83e16f\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") "
Oct 30 23:05:34.965286 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:34.964968    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-data\") pod \"e86e2368-d03b-4634-8245-eddf9d83e16f\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") "
Oct 30 23:05:34.965286 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:34.964992    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-credential-socket\") pod \"e86e2368-d03b-4634-8245-eddf9d83e16f\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") "
Oct 30 23:05:34.965286 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:34.965010    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-workload-socket\") pod \"e86e2368-d03b-4634-8245-eddf9d83e16f\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") "
Oct 30 23:05:34.965286 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:34.965035    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-token\" (UniqueName: \"kubernetes.io/projected/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-token\") pod \"e86e2368-d03b-4634-8245-eddf9d83e16f\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") "
Oct 30 23:05:34.965286 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:34.965063    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-workload-certs\") pod \"e86e2368-d03b-4634-8245-eddf9d83e16f\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") "
Oct 30 23:05:34.965286 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:34.965168    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-envoy\") pod \"e86e2368-d03b-4634-8245-eddf9d83e16f\" (UID: \"e86e2368-d03b-4634-8245-eddf9d83e16f\") "
Oct 30 23:05:34.965619 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:34.965347    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e86e2368-d03b-4634-8245-eddf9d83e16f-istiod-ca-cert" pod "e86e2368-d03b-4634-8245-eddf9d83e16f" (UID: "e86e2368-d03b-4634-8245-eddf9d83e16f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:05:34.965619 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:34.965472    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-workload-certs" pod "e86e2368-d03b-4634-8245-eddf9d83e16f" (UID: "e86e2368-d03b-4634-8245-eddf9d83e16f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:34.965619 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:34.965453    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-credential-socket" pod "e86e2368-d03b-4634-8245-eddf9d83e16f" (UID: "e86e2368-d03b-4634-8245-eddf9d83e16f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:34.965948 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:34.965679    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-workload-socket" pod "e86e2368-d03b-4634-8245-eddf9d83e16f" (UID: "e86e2368-d03b-4634-8245-eddf9d83e16f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:34.965948 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:34.965860    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-data" pod "e86e2368-d03b-4634-8245-eddf9d83e16f" (UID: "e86e2368-d03b-4634-8245-eddf9d83e16f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:34.967568 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:34.967541    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e86e2368-d03b-4634-8245-eddf9d83e16f-kube-api-access-49l74" pod "e86e2368-d03b-4634-8245-eddf9d83e16f" (UID: "e86e2368-d03b-4634-8245-eddf9d83e16f"). InnerVolumeSpecName "kube-api-access-49l74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:34.967705 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:34.967665    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-podinfo" pod "e86e2368-d03b-4634-8245-eddf9d83e16f" (UID: "e86e2368-d03b-4634-8245-eddf9d83e16f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:05:34.967759 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:34.967698    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-token" pod "e86e2368-d03b-4634-8245-eddf9d83e16f" (UID: "e86e2368-d03b-4634-8245-eddf9d83e16f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:34.967876 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:34.967850    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-envoy" pod "e86e2368-d03b-4634-8245-eddf9d83e16f" (UID: "e86e2368-d03b-4634-8245-eddf9d83e16f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:35.066904 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:35.066865    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:05:35.066904 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:35.066900    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e86e2368-d03b-4634-8245-eddf9d83e16f-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:05:35.066904 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:35.066909    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49l74\" (UniqueName: \"kubernetes.io/projected/e86e2368-d03b-4634-8245-eddf9d83e16f-kube-api-access-49l74\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:05:35.067157 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:35.066919    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:05:35.067157 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:35.066928    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:05:35.067157 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:35.066937    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:05:35.067157 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:35.066947    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:05:35.067157 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:35.066955    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e86e2368-d03b-4634-8245-eddf9d83e16f-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:05:35.067157 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:35.066963    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e86e2368-d03b-4634-8245-eddf9d83e16f-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:05:35.203446 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:35.203408    4889 generic.go:356] "Generic (PLEG): container finished" podID="e86e2368-d03b-4634-8245-eddf9d83e16f" containerID="2251e712dfc7caa34dcbd297d00dec6b23381762077159b56f5b7a1b1651ecfe" exitCode=0
Oct 30 23:05:35.203647 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:35.203472    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"
Oct 30 23:05:35.203647 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:35.203465    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k" event={"ID":"e86e2368-d03b-4634-8245-eddf9d83e16f","Type":"ContainerDied","Data":"2251e712dfc7caa34dcbd297d00dec6b23381762077159b56f5b7a1b1651ecfe"}
Oct 30 23:05:35.203647 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:35.203591    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k" event={"ID":"e86e2368-d03b-4634-8245-eddf9d83e16f","Type":"ContainerDied","Data":"adbdb77d559a01dc215ae4f6a2b25fd80795906954ef2c2bd757339ae41cfdfa"}
Oct 30 23:05:35.203647 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:35.203617    4889 scope.go:122] "RemoveContainer" containerID="2251e712dfc7caa34dcbd297d00dec6b23381762077159b56f5b7a1b1651ecfe"
Oct 30 23:05:35.213881 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:35.213863    4889 scope.go:122] "RemoveContainer" containerID="dd26b8faf73cb1e70647d9b1340ea734bcba1abca88c079ade95ec959d956c59"
Oct 30 23:05:35.222132 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:35.222102    4889 scope.go:122] "RemoveContainer" containerID="2251e712dfc7caa34dcbd297d00dec6b23381762077159b56f5b7a1b1651ecfe"
Oct 30 23:05:35.222409 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:05:35.222391    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2251e712dfc7caa34dcbd297d00dec6b23381762077159b56f5b7a1b1651ecfe\": container with ID starting with 2251e712dfc7caa34dcbd297d00dec6b23381762077159b56f5b7a1b1651ecfe not found: ID does not exist" containerID="2251e712dfc7caa34dcbd297d00dec6b23381762077159b56f5b7a1b1651ecfe"
Oct 30 23:05:35.222465 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:35.222418    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2251e712dfc7caa34dcbd297d00dec6b23381762077159b56f5b7a1b1651ecfe"} err="failed to get container status \"2251e712dfc7caa34dcbd297d00dec6b23381762077159b56f5b7a1b1651ecfe\": rpc error: code = NotFound desc = could not find container \"2251e712dfc7caa34dcbd297d00dec6b23381762077159b56f5b7a1b1651ecfe\": container with ID starting with 2251e712dfc7caa34dcbd297d00dec6b23381762077159b56f5b7a1b1651ecfe not found: ID does not exist"
Oct 30 23:05:35.222465 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:35.222439    4889 scope.go:122] "RemoveContainer" containerID="dd26b8faf73cb1e70647d9b1340ea734bcba1abca88c079ade95ec959d956c59"
Oct 30 23:05:35.222687 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:05:35.222669    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd26b8faf73cb1e70647d9b1340ea734bcba1abca88c079ade95ec959d956c59\": container with ID starting with dd26b8faf73cb1e70647d9b1340ea734bcba1abca88c079ade95ec959d956c59 not found: ID does not exist" containerID="dd26b8faf73cb1e70647d9b1340ea734bcba1abca88c079ade95ec959d956c59"
Oct 30 23:05:35.222744 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:35.222696    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd26b8faf73cb1e70647d9b1340ea734bcba1abca88c079ade95ec959d956c59"} err="failed to get container status \"dd26b8faf73cb1e70647d9b1340ea734bcba1abca88c079ade95ec959d956c59\": rpc error: code = NotFound desc = could not find container \"dd26b8faf73cb1e70647d9b1340ea734bcba1abca88c079ade95ec959d956c59\": container with ID starting with dd26b8faf73cb1e70647d9b1340ea734bcba1abca88c079ade95ec959d956c59 not found: ID does not exist"
Oct 30 23:05:35.259202 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:35.259157    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"]
Oct 30 23:05:35.260145 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:35.260111    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:35.272293 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:35.272262    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e14e00-openshift-default-565cddf77c-7jg4k"]
Oct 30 23:05:35.463210 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:35.463168    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:36.772655 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:36.772614    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e86e2368-d03b-4634-8245-eddf9d83e16f" path="/var/lib/kubelet/pods/e86e2368-d03b-4634-8245-eddf9d83e16f/volumes"
Oct 30 23:05:37.214810 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:37.214723    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:39.102987 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:39.102947    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:40.301447 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:05:40.301410    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:05:40.790906 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:40.790868    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx"]
Oct 30 23:05:40.942018 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:40.941982    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:41.129298 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:41.129197    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:42.394157 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394121    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c55f8ff-vqddm"]
Oct 30 23:05:42.394618 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394504    4889 cpu_manager.go:414] "containerMap: removing container" podUID="d893ba75-1d6a-404b-be7d-8b78ca3e6c62" containerName="revoke-keys"
Oct 30 23:05:42.394618 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394515    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d893ba75-1d6a-404b-be7d-8b78ca3e6c62" containerName="revoke-keys"
Oct 30 23:05:42.394618 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394524    4889 cpu_manager.go:414] "containerMap: removing container" podUID="e86e2368-d03b-4634-8245-eddf9d83e16f" containerName="istio-proxy"
Oct 30 23:05:42.394618 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394530    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e86e2368-d03b-4634-8245-eddf9d83e16f" containerName="istio-proxy"
Oct 30 23:05:42.394618 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394537    4889 cpu_manager.go:414] "containerMap: removing container" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main"
Oct 30 23:05:42.394618 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394543    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main"
Oct 30 23:05:42.394618 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394560    4889 cpu_manager.go:414] "containerMap: removing container" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main"
Oct 30 23:05:42.394618 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394565    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main"
Oct 30 23:05:42.394618 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394574    4889 cpu_manager.go:414] "containerMap: removing container" podUID="44c69100-209c-49c3-9c84-cb2d36028de5" containerName="maas-api"
Oct 30 23:05:42.394618 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394579    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c69100-209c-49c3-9c84-cb2d36028de5" containerName="maas-api"
Oct 30 23:05:42.394618 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394588    4889 cpu_manager.go:414] "containerMap: removing container" podUID="4c8f8098-b597-45ce-ac4b-60aa0910e60a" containerName="revoke-keys"
Oct 30 23:05:42.394618 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394593    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c8f8098-b597-45ce-ac4b-60aa0910e60a" containerName="revoke-keys"
Oct 30 23:05:42.394618 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394600    4889 cpu_manager.go:414] "containerMap: removing container" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main"
Oct 30 23:05:42.394618 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394604    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main"
Oct 30 23:05:42.394618 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394620    4889 cpu_manager.go:414] "containerMap: removing container" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main"
Oct 30 23:05:42.394618 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394624    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main"
Oct 30 23:05:42.395198 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394635    4889 cpu_manager.go:414] "containerMap: removing container" podUID="b69c09df-8ced-418f-92f4-6c9ead312931" containerName="istio-proxy"
Oct 30 23:05:42.395198 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394640    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b69c09df-8ced-418f-92f4-6c9ead312931" containerName="istio-proxy"
Oct 30 23:05:42.395198 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394650    4889 cpu_manager.go:414] "containerMap: removing container" podUID="e1595760-0ede-4e19-ad58-741f6f577207" containerName="payload-processing"
Oct 30 23:05:42.395198 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394655    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1595760-0ede-4e19-ad58-741f6f577207" containerName="payload-processing"
Oct 30 23:05:42.395198 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394707    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="009fde76-6593-492a-8c91-b566aa3f327a" containerName="main"
Oct 30 23:05:42.395198 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394714    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="44c69100-209c-49c3-9c84-cb2d36028de5" containerName="maas-api"
Oct 30 23:05:42.395198 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394723    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c8f8098-b597-45ce-ac4b-60aa0910e60a" containerName="revoke-keys"
Oct 30 23:05:42.395198 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394728    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="6af5c6ee-e62b-4095-924b-4d4440b91a26" containerName="main"
Oct 30 23:05:42.395198 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394736    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e99cfe5-3268-401d-b985-7634e2bac80b" containerName="main"
Oct 30 23:05:42.395198 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394742    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="e86e2368-d03b-4634-8245-eddf9d83e16f" containerName="istio-proxy"
Oct 30 23:05:42.395198 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394750    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="b69c09df-8ced-418f-92f4-6c9ead312931" containerName="istio-proxy"
Oct 30 23:05:42.395198 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394755    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1595760-0ede-4e19-ad58-741f6f577207" containerName="payload-processing"
Oct 30 23:05:42.395198 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394761    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="d893ba75-1d6a-404b-be7d-8b78ca3e6c62" containerName="revoke-keys"
Oct 30 23:05:42.395198 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394766    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="e86e2368-d03b-4634-8245-eddf9d83e16f" containerName="istio-proxy"
Oct 30 23:05:42.395198 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.394771    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d18c6ee-2f7b-4a85-9065-6c925004973d" containerName="main"
Oct 30 23:05:42.399170 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.399148    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c55f8ff-vqddm"
Oct 30 23:05:42.413973 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.413947    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c55f8ff-vqddm"]
Oct 30 23:05:42.498850 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.498812    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:05:42.499040 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.498858    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:05:42.536435 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.536392    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6b5ac7-8fa1-4443-840e-4b75885f0bad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c55f8ff-vqddm\" (UID: \"2b6b5ac7-8fa1-4443-840e-4b75885f0bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c55f8ff-vqddm"
Oct 30 23:05:42.536435 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.536442    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6b5ac7-8fa1-4443-840e-4b75885f0bad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c55f8ff-vqddm\" (UID: \"2b6b5ac7-8fa1-4443-840e-4b75885f0bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c55f8ff-vqddm"
Oct 30 23:05:42.536659 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.536552    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn67t\" (UniqueName: \"kubernetes.io/projected/2b6b5ac7-8fa1-4443-840e-4b75885f0bad-kube-api-access-nn67t\") pod \"maas-api-revoke-sub-3c55f8ff-vqddm\" (UID: \"2b6b5ac7-8fa1-4443-840e-4b75885f0bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c55f8ff-vqddm"
Oct 30 23:05:42.637953 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.637914    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6b5ac7-8fa1-4443-840e-4b75885f0bad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c55f8ff-vqddm\" (UID: \"2b6b5ac7-8fa1-4443-840e-4b75885f0bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c55f8ff-vqddm"
Oct 30 23:05:42.638176 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.637962    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6b5ac7-8fa1-4443-840e-4b75885f0bad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c55f8ff-vqddm\" (UID: \"2b6b5ac7-8fa1-4443-840e-4b75885f0bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c55f8ff-vqddm"
Oct 30 23:05:42.638176 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.638003    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn67t\" (UniqueName: \"kubernetes.io/projected/2b6b5ac7-8fa1-4443-840e-4b75885f0bad-kube-api-access-nn67t\") pod \"maas-api-revoke-sub-3c55f8ff-vqddm\" (UID: \"2b6b5ac7-8fa1-4443-840e-4b75885f0bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c55f8ff-vqddm"
Oct 30 23:05:42.638622 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.638599    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6b5ac7-8fa1-4443-840e-4b75885f0bad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c55f8ff-vqddm\" (UID: \"2b6b5ac7-8fa1-4443-840e-4b75885f0bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c55f8ff-vqddm"
Oct 30 23:05:42.640613 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.640588    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6b5ac7-8fa1-4443-840e-4b75885f0bad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c55f8ff-vqddm\" (UID: \"2b6b5ac7-8fa1-4443-840e-4b75885f0bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c55f8ff-vqddm"
Oct 30 23:05:42.659537 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.659470    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn67t\" (UniqueName: \"kubernetes.io/projected/2b6b5ac7-8fa1-4443-840e-4b75885f0bad-kube-api-access-nn67t\") pod \"maas-api-revoke-sub-3c55f8ff-vqddm\" (UID: \"2b6b5ac7-8fa1-4443-840e-4b75885f0bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c55f8ff-vqddm"
Oct 30 23:05:42.714305 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:42.714271    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c55f8ff-vqddm"
Oct 30 23:05:43.012440 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:43.012408    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c55f8ff-vqddm"]
Oct 30 23:05:43.017268 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:05:43.013826    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6b5ac7_8fa1_4443_840e_4b75885f0bad.slice/crio-df5c9df23cd8c1688e780cf9cd6fb2d188f777464c57896db4a08209272eec8d WatchSource:0}: Error finding container df5c9df23cd8c1688e780cf9cd6fb2d188f777464c57896db4a08209272eec8d: Status 404 returned error can't find the container with id df5c9df23cd8c1688e780cf9cd6fb2d188f777464c57896db4a08209272eec8d
Oct 30 23:05:43.242775 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:43.242741    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c55f8ff-vqddm" event={"ID":"2b6b5ac7-8fa1-4443-840e-4b75885f0bad","Type":"ContainerStarted","Data":"df5c9df23cd8c1688e780cf9cd6fb2d188f777464c57896db4a08209272eec8d"}
Oct 30 23:05:44.257364 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:44.257327    4889 generic.go:356] "Generic (PLEG): container finished" podID="2b6b5ac7-8fa1-4443-840e-4b75885f0bad" containerID="44c5d951f8b9234d29299624ce02606c10c4d957dc88e407068890b203271af1" exitCode=0
Oct 30 23:05:44.257793 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:44.257419    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c55f8ff-vqddm" event={"ID":"2b6b5ac7-8fa1-4443-840e-4b75885f0bad","Type":"ContainerDied","Data":"44c5d951f8b9234d29299624ce02606c10c4d957dc88e407068890b203271af1"}
Oct 30 23:05:45.398901 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:45.398870    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c55f8ff-vqddm"
Oct 30 23:05:45.466947 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:45.466911    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b6b5ac7-8fa1-4443-840e-4b75885f0bad-kube-api-access-nn67t\" (UniqueName: \"kubernetes.io/projected/2b6b5ac7-8fa1-4443-840e-4b75885f0bad-kube-api-access-nn67t\") pod \"2b6b5ac7-8fa1-4443-840e-4b75885f0bad\" (UID: \"2b6b5ac7-8fa1-4443-840e-4b75885f0bad\") "
Oct 30 23:05:45.467134 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:45.467006    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b6b5ac7-8fa1-4443-840e-4b75885f0bad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6b5ac7-8fa1-4443-840e-4b75885f0bad-maas-api-xks-ca\") pod \"2b6b5ac7-8fa1-4443-840e-4b75885f0bad\" (UID: \"2b6b5ac7-8fa1-4443-840e-4b75885f0bad\") "
Oct 30 23:05:45.467134 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:45.467110    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b6b5ac7-8fa1-4443-840e-4b75885f0bad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6b5ac7-8fa1-4443-840e-4b75885f0bad-maas-api-openshift-ca\") pod \"2b6b5ac7-8fa1-4443-840e-4b75885f0bad\" (UID: \"2b6b5ac7-8fa1-4443-840e-4b75885f0bad\") "
Oct 30 23:05:45.467495 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:45.467462    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6b5ac7-8fa1-4443-840e-4b75885f0bad-maas-api-openshift-ca" pod "2b6b5ac7-8fa1-4443-840e-4b75885f0bad" (UID: "2b6b5ac7-8fa1-4443-840e-4b75885f0bad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:05:45.469212 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:45.469191    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b6b5ac7-8fa1-4443-840e-4b75885f0bad-maas-api-xks-ca" pod "2b6b5ac7-8fa1-4443-840e-4b75885f0bad" (UID: "2b6b5ac7-8fa1-4443-840e-4b75885f0bad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:05:45.469338 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:45.469236    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6b5ac7-8fa1-4443-840e-4b75885f0bad-kube-api-access-nn67t" pod "2b6b5ac7-8fa1-4443-840e-4b75885f0bad" (UID: "2b6b5ac7-8fa1-4443-840e-4b75885f0bad"). InnerVolumeSpecName "kube-api-access-nn67t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:45.567944 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:45.567841    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6b5ac7-8fa1-4443-840e-4b75885f0bad-maas-api-xks-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:05:45.567944 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:45.567879    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6b5ac7-8fa1-4443-840e-4b75885f0bad-maas-api-openshift-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:05:45.567944 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:45.567894    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn67t\" (UniqueName: \"kubernetes.io/projected/2b6b5ac7-8fa1-4443-840e-4b75885f0bad-kube-api-access-nn67t\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:05:45.792069 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:45.792026    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx" podUID="5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0" containerName="payload-pre-processing" containerID="cri-o://1ca05f1cdfe7127647219bc1e430016dede4b8acfd227ad52c48c37c5e3fce39" gracePeriod=25
Oct 30 23:05:46.269737 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:46.269702    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c55f8ff-vqddm"
Oct 30 23:05:46.269928 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:46.269737    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c55f8ff-vqddm" event={"ID":"2b6b5ac7-8fa1-4443-840e-4b75885f0bad","Type":"ContainerDied","Data":"df5c9df23cd8c1688e780cf9cd6fb2d188f777464c57896db4a08209272eec8d"}
Oct 30 23:05:46.269928 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:46.269785    4889 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df5c9df23cd8c1688e780cf9cd6fb2d188f777464c57896db4a08209272eec8d"
Oct 30 23:05:47.410394 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:47.410357    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx" podUID="5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:05:51.961951 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:51.961912    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:57.410309 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:57.410271    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx" podUID="5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:05:57.630384 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:57.630342    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:57.666523 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:57.666447    4889 scope.go:122] "RemoveContainer" containerID="4ff87db5ab76260f525f3e4b56e1c424a067f4665ed892424bc0fc6faefc81da"
Oct 30 23:05:58.123678 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:58.123643    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:58.286210 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:58.286171    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:58.774427 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:05:58.774390    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:06:05.384670 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:05.384576    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:06:05.588926 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:05.588878    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:06:05.756543 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:05.756508    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:06:06.039584 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:06.039558    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx"
Oct 30 23:06:06.050217 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:06.050187    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0-kube-api-access-z79gk\" (UniqueName: \"kubernetes.io/projected/5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0-kube-api-access-z79gk\") pod \"5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0\" (UID: \"5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0\") "
Oct 30 23:06:06.050404 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:06.050320    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0-plugins-config-volume\") pod \"5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0\" (UID: \"5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0\") "
Oct 30 23:06:06.050685 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:06.050661    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0-plugins-config-volume" pod "5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0" (UID: "5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:06:06.052507 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:06.052486    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0-kube-api-access-z79gk" pod "5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0" (UID: "5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0"). InnerVolumeSpecName "kube-api-access-z79gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:06:06.151783 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:06.151744    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:06:06.151783 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:06.151778    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z79gk\" (UniqueName: \"kubernetes.io/projected/5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0-kube-api-access-z79gk\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:06:06.369399 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:06.369313    4889 generic.go:356] "Generic (PLEG): container finished" podID="5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0" containerID="1ca05f1cdfe7127647219bc1e430016dede4b8acfd227ad52c48c37c5e3fce39" exitCode=0
Oct 30 23:06:06.369399 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:06.369366    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx"
Oct 30 23:06:06.369623 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:06.369393    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx" event={"ID":"5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0","Type":"ContainerDied","Data":"1ca05f1cdfe7127647219bc1e430016dede4b8acfd227ad52c48c37c5e3fce39"}
Oct 30 23:06:06.369623 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:06.369452    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx" event={"ID":"5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0","Type":"ContainerDied","Data":"9c94dbc84b755913122d7ea5607de5e1f04f9dd81e0ed201de497d255b9d5a10"}
Oct 30 23:06:06.369623 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:06.369482    4889 scope.go:122] "RemoveContainer" containerID="1ca05f1cdfe7127647219bc1e430016dede4b8acfd227ad52c48c37c5e3fce39"
Oct 30 23:06:06.378980 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:06.378958    4889 scope.go:122] "RemoveContainer" containerID="1ca05f1cdfe7127647219bc1e430016dede4b8acfd227ad52c48c37c5e3fce39"
Oct 30 23:06:06.379292 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:06:06.379271    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ca05f1cdfe7127647219bc1e430016dede4b8acfd227ad52c48c37c5e3fce39\": container with ID starting with 1ca05f1cdfe7127647219bc1e430016dede4b8acfd227ad52c48c37c5e3fce39 not found: ID does not exist" containerID="1ca05f1cdfe7127647219bc1e430016dede4b8acfd227ad52c48c37c5e3fce39"
Oct 30 23:06:06.379353 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:06.379305    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca05f1cdfe7127647219bc1e430016dede4b8acfd227ad52c48c37c5e3fce39"} err="failed to get container status \"1ca05f1cdfe7127647219bc1e430016dede4b8acfd227ad52c48c37c5e3fce39\": rpc error: code = NotFound desc = could not find container \"1ca05f1cdfe7127647219bc1e430016dede4b8acfd227ad52c48c37c5e3fce39\": container with ID starting with 1ca05f1cdfe7127647219bc1e430016dede4b8acfd227ad52c48c37c5e3fce39 not found: ID does not exist"
Oct 30 23:06:06.425081 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:06.425052    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx"]
Oct 30 23:06:06.438889 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:06.438864    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d52c59-56d5c877c5-zvrjx"]
Oct 30 23:06:06.772942 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:06.772912    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0" path="/var/lib/kubelet/pods/5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0/volumes"
Oct 30 23:06:10.302117 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:06:10.302082    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:06:10.432137 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:10.432104    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8586c445d6-g6ktk"]
Oct 30 23:06:10.432369 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:10.432332    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-8586c445d6-g6ktk" podUID="347d81d0-5e87-42c4-89e7-8666cab18419" containerName="manager" containerID="cri-o://ee9dd4f65be63b30e3e2e7f58ed50852dc71cf32d25f2955fac0f99265c5d5e9" gracePeriod=30
Oct 30 23:06:10.664139 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:10.664113    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8586c445d6-g6ktk"
Oct 30 23:06:10.690481 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:10.690453    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/347d81d0-5e87-42c4-89e7-8666cab18419-webhook-cert\" (UniqueName: \"kubernetes.io/secret/347d81d0-5e87-42c4-89e7-8666cab18419-webhook-cert\") pod \"347d81d0-5e87-42c4-89e7-8666cab18419\" (UID: \"347d81d0-5e87-42c4-89e7-8666cab18419\") "
Oct 30 23:06:10.690640 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:10.690515    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/347d81d0-5e87-42c4-89e7-8666cab18419-metrics-certs\" (UniqueName: \"kubernetes.io/secret/347d81d0-5e87-42c4-89e7-8666cab18419-metrics-certs\") pod \"347d81d0-5e87-42c4-89e7-8666cab18419\" (UID: \"347d81d0-5e87-42c4-89e7-8666cab18419\") "
Oct 30 23:06:10.690640 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:10.690559    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/347d81d0-5e87-42c4-89e7-8666cab18419-kube-api-access-fsc9s\" (UniqueName: \"kubernetes.io/projected/347d81d0-5e87-42c4-89e7-8666cab18419-kube-api-access-fsc9s\") pod \"347d81d0-5e87-42c4-89e7-8666cab18419\" (UID: \"347d81d0-5e87-42c4-89e7-8666cab18419\") "
Oct 30 23:06:10.692835 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:10.692804    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/347d81d0-5e87-42c4-89e7-8666cab18419-webhook-cert" pod "347d81d0-5e87-42c4-89e7-8666cab18419" (UID: "347d81d0-5e87-42c4-89e7-8666cab18419"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:06:10.692835 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:10.692816    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/347d81d0-5e87-42c4-89e7-8666cab18419-metrics-certs" pod "347d81d0-5e87-42c4-89e7-8666cab18419" (UID: "347d81d0-5e87-42c4-89e7-8666cab18419"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:06:10.692835 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:10.692824    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347d81d0-5e87-42c4-89e7-8666cab18419-kube-api-access-fsc9s" pod "347d81d0-5e87-42c4-89e7-8666cab18419" (UID: "347d81d0-5e87-42c4-89e7-8666cab18419"). InnerVolumeSpecName "kube-api-access-fsc9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:06:10.791460 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:10.791433    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/347d81d0-5e87-42c4-89e7-8666cab18419-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:06:10.791460 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:10.791459    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsc9s\" (UniqueName: \"kubernetes.io/projected/347d81d0-5e87-42c4-89e7-8666cab18419-kube-api-access-fsc9s\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:06:10.791679 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:10.791468    4889 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/347d81d0-5e87-42c4-89e7-8666cab18419-webhook-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:06:11.395105 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:11.395067    4889 generic.go:356] "Generic (PLEG): container finished" podID="347d81d0-5e87-42c4-89e7-8666cab18419" containerID="ee9dd4f65be63b30e3e2e7f58ed50852dc71cf32d25f2955fac0f99265c5d5e9" exitCode=0
Oct 30 23:06:11.395592 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:11.395114    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8586c445d6-g6ktk"
Oct 30 23:06:11.395592 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:11.395155    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8586c445d6-g6ktk" event={"ID":"347d81d0-5e87-42c4-89e7-8666cab18419","Type":"ContainerDied","Data":"ee9dd4f65be63b30e3e2e7f58ed50852dc71cf32d25f2955fac0f99265c5d5e9"}
Oct 30 23:06:11.395592 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:11.395208    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8586c445d6-g6ktk" event={"ID":"347d81d0-5e87-42c4-89e7-8666cab18419","Type":"ContainerDied","Data":"4a47c1d268a170f167f2dfd1ae76d7d60df2f349a7ba460f0e027701b50718ed"}
Oct 30 23:06:11.395592 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:11.395255    4889 scope.go:122] "RemoveContainer" containerID="ee9dd4f65be63b30e3e2e7f58ed50852dc71cf32d25f2955fac0f99265c5d5e9"
Oct 30 23:06:11.405590 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:11.405569    4889 scope.go:122] "RemoveContainer" containerID="ee9dd4f65be63b30e3e2e7f58ed50852dc71cf32d25f2955fac0f99265c5d5e9"
Oct 30 23:06:11.405856 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:06:11.405830    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee9dd4f65be63b30e3e2e7f58ed50852dc71cf32d25f2955fac0f99265c5d5e9\": container with ID starting with ee9dd4f65be63b30e3e2e7f58ed50852dc71cf32d25f2955fac0f99265c5d5e9 not found: ID does not exist" containerID="ee9dd4f65be63b30e3e2e7f58ed50852dc71cf32d25f2955fac0f99265c5d5e9"
Oct 30 23:06:11.405927 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:11.405870    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee9dd4f65be63b30e3e2e7f58ed50852dc71cf32d25f2955fac0f99265c5d5e9"} err="failed to get container status \"ee9dd4f65be63b30e3e2e7f58ed50852dc71cf32d25f2955fac0f99265c5d5e9\": rpc error: code = NotFound desc = could not find container \"ee9dd4f65be63b30e3e2e7f58ed50852dc71cf32d25f2955fac0f99265c5d5e9\": container with ID starting with ee9dd4f65be63b30e3e2e7f58ed50852dc71cf32d25f2955fac0f99265c5d5e9 not found: ID does not exist"
Oct 30 23:06:11.450698 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:11.450664    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8586c445d6-g6ktk"]
Oct 30 23:06:11.464101 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:11.464072    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8586c445d6-g6ktk"]
Oct 30 23:06:11.968355 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:11.968317    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:06:12.157159 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:12.157120    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:06:12.357585 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:12.357553    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:06:12.564866 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:12.564816    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:06:12.772741 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:12.772709    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="347d81d0-5e87-42c4-89e7-8666cab18419" path="/var/lib/kubelet/pods/347d81d0-5e87-42c4-89e7-8666cab18419/volumes"
Oct 30 23:06:12.856366 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:12.856328    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8586c445d6-7dkgj"]
Oct 30 23:06:12.856720 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:12.856706    4889 cpu_manager.go:414] "containerMap: removing container" podUID="347d81d0-5e87-42c4-89e7-8666cab18419" containerName="manager"
Oct 30 23:06:12.856780 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:12.856722    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="347d81d0-5e87-42c4-89e7-8666cab18419" containerName="manager"
Oct 30 23:06:12.856780 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:12.856744    4889 cpu_manager.go:414] "containerMap: removing container" podUID="5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0" containerName="payload-pre-processing"
Oct 30 23:06:12.856780 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:12.856750    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0" containerName="payload-pre-processing"
Oct 30 23:06:12.856780 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:12.856758    4889 cpu_manager.go:414] "containerMap: removing container" podUID="e86e2368-d03b-4634-8245-eddf9d83e16f" containerName="istio-proxy"
Oct 30 23:06:12.856780 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:12.856763    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e86e2368-d03b-4634-8245-eddf9d83e16f" containerName="istio-proxy"
Oct 30 23:06:12.856780 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:12.856782    4889 cpu_manager.go:414] "containerMap: removing container" podUID="2b6b5ac7-8fa1-4443-840e-4b75885f0bad" containerName="revoke-keys"
Oct 30 23:06:12.856978 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:12.856788    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b6b5ac7-8fa1-4443-840e-4b75885f0bad" containerName="revoke-keys"
Oct 30 23:06:12.856978 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:12.856843    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="347d81d0-5e87-42c4-89e7-8666cab18419" containerName="manager"
Oct 30 23:06:12.856978 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:12.856853    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cb47a7e-8385-4154-a3b2-8ff2e1ae39b0" containerName="payload-pre-processing"
Oct 30 23:06:12.856978 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:12.856863    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b6b5ac7-8fa1-4443-840e-4b75885f0bad" containerName="revoke-keys"
Oct 30 23:06:12.860044 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:12.860028    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8586c445d6-7dkgj"
Oct 30 23:06:12.890381 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:12.890336    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 23:06:12.890381 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:12.890367    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 23:06:12.890381 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:12.890351    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jthrb\""
Oct 30 23:06:12.892068 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:12.892042    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8586c445d6-7dkgj"]
Oct 30 23:06:12.910826 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:12.910792    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4b3939b-c274-47f1-a31f-82245276004f-metrics-certs\") pod \"maas-controller-8586c445d6-7dkgj\" (UID: \"c4b3939b-c274-47f1-a31f-82245276004f\") " pod="opendatahub/maas-controller-8586c445d6-7dkgj"
Oct 30 23:06:12.911045 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:12.910848    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g7wk\" (UniqueName: \"kubernetes.io/projected/c4b3939b-c274-47f1-a31f-82245276004f-kube-api-access-8g7wk\") pod \"maas-controller-8586c445d6-7dkgj\" (UID: \"c4b3939b-c274-47f1-a31f-82245276004f\") " pod="opendatahub/maas-controller-8586c445d6-7dkgj"
Oct 30 23:06:12.911045 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:12.910946    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4b3939b-c274-47f1-a31f-82245276004f-webhook-cert\") pod \"maas-controller-8586c445d6-7dkgj\" (UID: \"c4b3939b-c274-47f1-a31f-82245276004f\") " pod="opendatahub/maas-controller-8586c445d6-7dkgj"
Oct 30 23:06:12.946253 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:12.946204    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 23:06:13.011748 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:13.011705    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g7wk\" (UniqueName: \"kubernetes.io/projected/c4b3939b-c274-47f1-a31f-82245276004f-kube-api-access-8g7wk\") pod \"maas-controller-8586c445d6-7dkgj\" (UID: \"c4b3939b-c274-47f1-a31f-82245276004f\") " pod="opendatahub/maas-controller-8586c445d6-7dkgj"
Oct 30 23:06:13.011748 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:13.011753    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4b3939b-c274-47f1-a31f-82245276004f-webhook-cert\") pod \"maas-controller-8586c445d6-7dkgj\" (UID: \"c4b3939b-c274-47f1-a31f-82245276004f\") " pod="opendatahub/maas-controller-8586c445d6-7dkgj"
Oct 30 23:06:13.012021 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:13.011827    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4b3939b-c274-47f1-a31f-82245276004f-metrics-certs\") pod \"maas-controller-8586c445d6-7dkgj\" (UID: \"c4b3939b-c274-47f1-a31f-82245276004f\") " pod="opendatahub/maas-controller-8586c445d6-7dkgj"
Oct 30 23:06:13.014640 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:13.014605    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4b3939b-c274-47f1-a31f-82245276004f-webhook-cert\") pod \"maas-controller-8586c445d6-7dkgj\" (UID: \"c4b3939b-c274-47f1-a31f-82245276004f\") " pod="opendatahub/maas-controller-8586c445d6-7dkgj"
Oct 30 23:06:13.014782 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:13.014614    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4b3939b-c274-47f1-a31f-82245276004f-metrics-certs\") pod \"maas-controller-8586c445d6-7dkgj\" (UID: \"c4b3939b-c274-47f1-a31f-82245276004f\") " pod="opendatahub/maas-controller-8586c445d6-7dkgj"
Oct 30 23:06:13.033079 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:13.032988    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g7wk\" (UniqueName: \"kubernetes.io/projected/c4b3939b-c274-47f1-a31f-82245276004f-kube-api-access-8g7wk\") pod \"maas-controller-8586c445d6-7dkgj\" (UID: \"c4b3939b-c274-47f1-a31f-82245276004f\") " pod="opendatahub/maas-controller-8586c445d6-7dkgj"
Oct 30 23:06:13.173630 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:13.173590    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8586c445d6-7dkgj"
Oct 30 23:06:13.474702 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:13.474676    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8586c445d6-7dkgj"]
Oct 30 23:06:13.476029 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:06:13.476001    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b3939b_c274_47f1_a31f_82245276004f.slice/crio-4b6710aa0ef5850fbbe5bb9948c093c70366df47333087dd82b17509e692c74e WatchSource:0}: Error finding container 4b6710aa0ef5850fbbe5bb9948c093c70366df47333087dd82b17509e692c74e: Status 404 returned error can't find the container with id 4b6710aa0ef5850fbbe5bb9948c093c70366df47333087dd82b17509e692c74e
Oct 30 23:06:13.477660 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:13.477638    4889 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:06:13.522338 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:13.522291    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:06:14.412039 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:14.412002    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8586c445d6-7dkgj" event={"ID":"c4b3939b-c274-47f1-a31f-82245276004f","Type":"ContainerStarted","Data":"4ac1905c8dc19bafc5d71f5abd43b7b4bf7894e66676d8e0055c75c9702302f1"}
Oct 30 23:06:14.412522 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:14.412063    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8586c445d6-7dkgj" event={"ID":"c4b3939b-c274-47f1-a31f-82245276004f","Type":"ContainerStarted","Data":"4b6710aa0ef5850fbbe5bb9948c093c70366df47333087dd82b17509e692c74e"}
Oct 30 23:06:14.412522 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:14.412092    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8586c445d6-7dkgj"
Oct 30 23:06:14.455489 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:14.455436    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-8586c445d6-7dkgj" podStartSLOduration=1.980801417 podStartE2EDuration="2.455422364s" totalImagesPullingTime="474.620947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:14.454988744 +0000 UTC m=+3559.811355059" watchObservedRunningTime="2026-10-30 23:06:14.455422364 +0000 UTC m=+3559.811788673"
Oct 30 23:06:19.340001 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:19.339961    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:06:25.420030 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:25.419997    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8586c445d6-7dkgj"
Oct 30 23:06:29.173763 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.173729    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"]
Oct 30 23:06:29.177460 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.177437    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.187910 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.187882    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"]
Oct 30 23:06:29.191038 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.191021    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.200187 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.200160    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 23:06:29.200334 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.200318    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-j98cq\""
Oct 30 23:06:29.202405 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.202381    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"]
Oct 30 23:06:29.211729 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.211661    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"]
Oct 30 23:06:29.255339 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.255306    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fa4b71-d6b5-4cd7-b768-6c87b5593a7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p2flk\" (UID: \"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.255339 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.255349    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8d3e5a5-6b17-4204-bd2e-5099505f1bc4-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-v8xjj\" (UID: \"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.255584 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.255367    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kwkv\" (UniqueName: \"kubernetes.io/projected/63fa4b71-d6b5-4cd7-b768-6c87b5593a7c-kube-api-access-2kwkv\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p2flk\" (UID: \"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.255584 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.255425    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fa4b71-d6b5-4cd7-b768-6c87b5593a7c-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p2flk\" (UID: \"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.255584 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.255462    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8d3e5a5-6b17-4204-bd2e-5099505f1bc4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-v8xjj\" (UID: \"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.255584 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.255481    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fa4b71-d6b5-4cd7-b768-6c87b5593a7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p2flk\" (UID: \"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.255584 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.255543    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8d3e5a5-6b17-4204-bd2e-5099505f1bc4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-v8xjj\" (UID: \"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.255584 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.255577    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7v2t\" (UniqueName: \"kubernetes.io/projected/c8d3e5a5-6b17-4204-bd2e-5099505f1bc4-kube-api-access-q7v2t\") pod \"maas-default-gateway-openshift-default-6db586dd6c-v8xjj\" (UID: \"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.255788 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.255608    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fa4b71-d6b5-4cd7-b768-6c87b5593a7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p2flk\" (UID: \"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.255788 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.255632    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fa4b71-d6b5-4cd7-b768-6c87b5593a7c-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p2flk\" (UID: \"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.255788 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.255649    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d3e5a5-6b17-4204-bd2e-5099505f1bc4-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-v8xjj\" (UID: \"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.255788 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.255669    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8d3e5a5-6b17-4204-bd2e-5099505f1bc4-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-v8xjj\" (UID: \"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.255788 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.255775    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fa4b71-d6b5-4cd7-b768-6c87b5593a7c-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p2flk\" (UID: \"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.255953 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.255812    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d3e5a5-6b17-4204-bd2e-5099505f1bc4-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-v8xjj\" (UID: \"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.255953 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.255831    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8d3e5a5-6b17-4204-bd2e-5099505f1bc4-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-v8xjj\" (UID: \"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.255953 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.255878    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63fa4b71-d6b5-4cd7-b768-6c87b5593a7c-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p2flk\" (UID: \"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.255953 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.255909    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fa4b71-d6b5-4cd7-b768-6c87b5593a7c-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p2flk\" (UID: \"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.255953 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.255928    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8d3e5a5-6b17-4204-bd2e-5099505f1bc4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-v8xjj\" (UID: \"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.357236 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.357194    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7v2t\" (UniqueName: \"kubernetes.io/projected/c8d3e5a5-6b17-4204-bd2e-5099505f1bc4-kube-api-access-q7v2t\") pod \"maas-default-gateway-openshift-default-6db586dd6c-v8xjj\" (UID: \"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.357435 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.357275    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fa4b71-d6b5-4cd7-b768-6c87b5593a7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p2flk\" (UID: \"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.357435 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.357296    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fa4b71-d6b5-4cd7-b768-6c87b5593a7c-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p2flk\" (UID: \"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.357435 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.357317    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d3e5a5-6b17-4204-bd2e-5099505f1bc4-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-v8xjj\" (UID: \"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.357435 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.357352    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8d3e5a5-6b17-4204-bd2e-5099505f1bc4-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-v8xjj\" (UID: \"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.357435 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.357408    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fa4b71-d6b5-4cd7-b768-6c87b5593a7c-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p2flk\" (UID: \"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.357728 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.357442    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d3e5a5-6b17-4204-bd2e-5099505f1bc4-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-v8xjj\" (UID: \"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.357728 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.357467    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8d3e5a5-6b17-4204-bd2e-5099505f1bc4-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-v8xjj\" (UID: \"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.357728 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.357491    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63fa4b71-d6b5-4cd7-b768-6c87b5593a7c-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p2flk\" (UID: \"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.357728 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.357516    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fa4b71-d6b5-4cd7-b768-6c87b5593a7c-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p2flk\" (UID: \"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.357728 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.357541    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8d3e5a5-6b17-4204-bd2e-5099505f1bc4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-v8xjj\" (UID: \"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.357728 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.357591    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fa4b71-d6b5-4cd7-b768-6c87b5593a7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p2flk\" (UID: \"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.357728 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.357639    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8d3e5a5-6b17-4204-bd2e-5099505f1bc4-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-v8xjj\" (UID: \"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.357728 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.357667    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kwkv\" (UniqueName: \"kubernetes.io/projected/63fa4b71-d6b5-4cd7-b768-6c87b5593a7c-kube-api-access-2kwkv\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p2flk\" (UID: \"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.357728 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.357695    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fa4b71-d6b5-4cd7-b768-6c87b5593a7c-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p2flk\" (UID: \"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.358272 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.357737    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8d3e5a5-6b17-4204-bd2e-5099505f1bc4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-v8xjj\" (UID: \"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.358272 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.357767    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fa4b71-d6b5-4cd7-b768-6c87b5593a7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p2flk\" (UID: \"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.358272 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.357798    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8d3e5a5-6b17-4204-bd2e-5099505f1bc4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-v8xjj\" (UID: \"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.358272 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.357824    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fa4b71-d6b5-4cd7-b768-6c87b5593a7c-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p2flk\" (UID: \"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.358272 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.357863    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d3e5a5-6b17-4204-bd2e-5099505f1bc4-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-v8xjj\" (UID: \"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.358272 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.357918    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d3e5a5-6b17-4204-bd2e-5099505f1bc4-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-v8xjj\" (UID: \"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.358272 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.357953    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8d3e5a5-6b17-4204-bd2e-5099505f1bc4-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-v8xjj\" (UID: \"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.358272 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.358108    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8d3e5a5-6b17-4204-bd2e-5099505f1bc4-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-v8xjj\" (UID: \"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.358272 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.358119    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fa4b71-d6b5-4cd7-b768-6c87b5593a7c-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p2flk\" (UID: \"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.358272 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.358245    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fa4b71-d6b5-4cd7-b768-6c87b5593a7c-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p2flk\" (UID: \"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.358817 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.358269    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fa4b71-d6b5-4cd7-b768-6c87b5593a7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p2flk\" (UID: \"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.358817 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.358559    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fa4b71-d6b5-4cd7-b768-6c87b5593a7c-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p2flk\" (UID: \"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.359115 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.359093    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8d3e5a5-6b17-4204-bd2e-5099505f1bc4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-v8xjj\" (UID: \"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.360485 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.360465    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fa4b71-d6b5-4cd7-b768-6c87b5593a7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p2flk\" (UID: \"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.360688 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.360660    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8d3e5a5-6b17-4204-bd2e-5099505f1bc4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-v8xjj\" (UID: \"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.360831 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.360809    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8d3e5a5-6b17-4204-bd2e-5099505f1bc4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-v8xjj\" (UID: \"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.361100 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.361080    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fa4b71-d6b5-4cd7-b768-6c87b5593a7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p2flk\" (UID: \"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.378974 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.378943    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63fa4b71-d6b5-4cd7-b768-6c87b5593a7c-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p2flk\" (UID: \"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.379083 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.379021    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kwkv\" (UniqueName: \"kubernetes.io/projected/63fa4b71-d6b5-4cd7-b768-6c87b5593a7c-kube-api-access-2kwkv\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p2flk\" (UID: \"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.379497 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.379480    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8d3e5a5-6b17-4204-bd2e-5099505f1bc4-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-v8xjj\" (UID: \"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.379564 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.379548    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7v2t\" (UniqueName: \"kubernetes.io/projected/c8d3e5a5-6b17-4204-bd2e-5099505f1bc4-kube-api-access-q7v2t\") pod \"maas-default-gateway-openshift-default-6db586dd6c-v8xjj\" (UID: \"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.490148 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.490048    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:29.504385 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.504356    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:29.823941 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.823875    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"]
Oct 30 23:06:29.824815 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:06:29.824791    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63fa4b71_d6b5_4cd7_b768_6c87b5593a7c.slice/crio-4647dde7d31de0b1a84ed2462d681438d397a2737c85ebd064fe87ee12b3379b WatchSource:0}: Error finding container 4647dde7d31de0b1a84ed2462d681438d397a2737c85ebd064fe87ee12b3379b: Status 404 returned error can't find the container with id 4647dde7d31de0b1a84ed2462d681438d397a2737c85ebd064fe87ee12b3379b
Oct 30 23:06:29.827473 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.827443    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:06:29.827592 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.827504    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:06:29.827592 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.827537    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:06:29.827879 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:06:29.827857    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d3e5a5_6b17_4204_bd2e_5099505f1bc4.slice/crio-6d570db75650dc352a124886ac5a2119b90e87d6ac6ad50f414e9fb763e7eb5c WatchSource:0}: Error finding container 6d570db75650dc352a124886ac5a2119b90e87d6ac6ad50f414e9fb763e7eb5c: Status 404 returned error can't find the container with id 6d570db75650dc352a124886ac5a2119b90e87d6ac6ad50f414e9fb763e7eb5c
Oct 30 23:06:29.830370 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.830341    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:06:29.830476 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.830401    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:06:29.830476 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.830430    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:06:29.837529 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:29.837503    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"]
Oct 30 23:06:30.490873 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:30.490831    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj" event={"ID":"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4","Type":"ContainerStarted","Data":"4f78901d64943633c2b9724dfbf7da0ea63bcfd541da4e5ac0f413573db128d4"}
Oct 30 23:06:30.491438 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:30.490907    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj" event={"ID":"c8d3e5a5-6b17-4204-bd2e-5099505f1bc4","Type":"ContainerStarted","Data":"6d570db75650dc352a124886ac5a2119b90e87d6ac6ad50f414e9fb763e7eb5c"}
Oct 30 23:06:30.492332 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:30.492304    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk" event={"ID":"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c","Type":"ContainerStarted","Data":"aafd3e50c5eec14992a756424623178bbf33fcd6cadfc0d3a76eb8a918b3faa7"}
Oct 30 23:06:30.492467 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:30.492348    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk" event={"ID":"63fa4b71-d6b5-4cd7-b768-6c87b5593a7c","Type":"ContainerStarted","Data":"4647dde7d31de0b1a84ed2462d681438d397a2737c85ebd064fe87ee12b3379b"}
Oct 30 23:06:30.504730 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:30.504707    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:30.506190 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:30.506169    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p2flk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Oct 30 23:06:30.506299 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:30.506211    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk" podUID="63fa4b71-d6b5-4cd7-b768-6c87b5593a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Oct 30 23:06:30.561004 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:30.560930    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj" podStartSLOduration=1.560910957 podStartE2EDuration="1.560910957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:30.556211738 +0000 UTC m=+3575.912578053" watchObservedRunningTime="2026-10-30 23:06:30.560910957 +0000 UTC m=+3575.917277277"
Oct 30 23:06:30.600963 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:30.600918    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk" podStartSLOduration=1.600904903 podStartE2EDuration="1.600904903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:30.600880268 +0000 UTC m=+3575.957246582" watchObservedRunningTime="2026-10-30 23:06:30.600904903 +0000 UTC m=+3575.957271214"
Oct 30 23:06:31.490983 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:31.490948    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:31.492734 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:31.492707    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-v8xjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Oct 30 23:06:31.492895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:31.492763    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj" podUID="c8d3e5a5-6b17-4204-bd2e-5099505f1bc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Oct 30 23:06:31.505668 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:31.505643    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p2flk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Oct 30 23:06:31.505791 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:31.505685    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk" podUID="63fa4b71-d6b5-4cd7-b768-6c87b5593a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Oct 30 23:06:32.491020 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:32.490979    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-v8xjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Oct 30 23:06:32.491483 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:32.491048    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj" podUID="c8d3e5a5-6b17-4204-bd2e-5099505f1bc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Oct 30 23:06:32.505768 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:32.505739    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p2flk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Oct 30 23:06:32.505913 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:32.505791    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk" podUID="63fa4b71-d6b5-4cd7-b768-6c87b5593a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Oct 30 23:06:33.491423 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:33.491374    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-v8xjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Oct 30 23:06:33.491891 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:33.491440    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj" podUID="c8d3e5a5-6b17-4204-bd2e-5099505f1bc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Oct 30 23:06:33.504836 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:33.504812    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p2flk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Oct 30 23:06:33.504970 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:33.504856    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk" podUID="63fa4b71-d6b5-4cd7-b768-6c87b5593a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Oct 30 23:06:34.490917 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:34.490879    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-v8xjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Oct 30 23:06:34.491141 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:34.490933    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj" podUID="c8d3e5a5-6b17-4204-bd2e-5099505f1bc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Oct 30 23:06:34.505202 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:34.505175    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p2flk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Oct 30 23:06:34.505582 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:34.505242    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk" podUID="63fa4b71-d6b5-4cd7-b768-6c87b5593a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Oct 30 23:06:35.490762 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:35.490715    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-v8xjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Oct 30 23:06:35.491043 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:35.490775    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj" podUID="c8d3e5a5-6b17-4204-bd2e-5099505f1bc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Oct 30 23:06:35.505111 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:35.505076    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p2flk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Oct 30 23:06:35.505256 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:35.505153    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk" podUID="63fa4b71-d6b5-4cd7-b768-6c87b5593a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Oct 30 23:06:36.491268 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:36.491213    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-v8xjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Oct 30 23:06:36.491507 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:36.491320    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj" podUID="c8d3e5a5-6b17-4204-bd2e-5099505f1bc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Oct 30 23:06:36.505791 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:36.505753    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p2flk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Oct 30 23:06:36.506172 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:36.505818    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk" podUID="63fa4b71-d6b5-4cd7-b768-6c87b5593a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Oct 30 23:06:37.490799 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:37.490759    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-v8xjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Oct 30 23:06:37.491007 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:37.490833    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj" podUID="c8d3e5a5-6b17-4204-bd2e-5099505f1bc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Oct 30 23:06:37.505148 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:37.505114    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p2flk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Oct 30 23:06:37.505294 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:37.505165    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk" podUID="63fa4b71-d6b5-4cd7-b768-6c87b5593a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Oct 30 23:06:39.491069 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:39.491024    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-v8xjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:06:39.491560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:39.491103    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj" podUID="c8d3e5a5-6b17-4204-bd2e-5099505f1bc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:06:39.545421 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:39.545369    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p2flk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:06:39.545638 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:39.545454    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk" podUID="63fa4b71-d6b5-4cd7-b768-6c87b5593a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:06:40.301444 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:06:40.301409    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:06:40.532382 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:40.532343    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-v8xjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:06:40.532934 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:40.532407    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj" podUID="c8d3e5a5-6b17-4204-bd2e-5099505f1bc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:06:40.577662 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:40.577572    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p2flk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:06:40.577662 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:40.577644    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk" podUID="63fa4b71-d6b5-4cd7-b768-6c87b5593a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:06:40.742847 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:40.742814    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:40.743735 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:40.743703    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:40.744731 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:40.744708    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk"
Oct 30 23:06:41.300521 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:41.300488    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:41.300827 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:41.300800    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:41.301433 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:41.301417    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj"
Oct 30 23:06:43.555311 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:43.555266    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:06:43.623461 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:43.623417    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8586c445d6-7dkgj"]
Oct 30 23:06:43.623690 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:43.623662    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-8586c445d6-7dkgj" podUID="c4b3939b-c274-47f1-a31f-82245276004f" containerName="manager" containerID="cri-o://4ac1905c8dc19bafc5d71f5abd43b7b4bf7894e66676d8e0055c75c9702302f1" gracePeriod=30
Oct 30 23:06:43.864722 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:43.864690    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8586c445d6-7dkgj"
Oct 30 23:06:44.048772 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:44.048741    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4b3939b-c274-47f1-a31f-82245276004f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4b3939b-c274-47f1-a31f-82245276004f-metrics-certs\") pod \"c4b3939b-c274-47f1-a31f-82245276004f\" (UID: \"c4b3939b-c274-47f1-a31f-82245276004f\") "
Oct 30 23:06:44.048946 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:44.048788    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4b3939b-c274-47f1-a31f-82245276004f-webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4b3939b-c274-47f1-a31f-82245276004f-webhook-cert\") pod \"c4b3939b-c274-47f1-a31f-82245276004f\" (UID: \"c4b3939b-c274-47f1-a31f-82245276004f\") "
Oct 30 23:06:44.048946 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:44.048832    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4b3939b-c274-47f1-a31f-82245276004f-kube-api-access-8g7wk\" (UniqueName: \"kubernetes.io/projected/c4b3939b-c274-47f1-a31f-82245276004f-kube-api-access-8g7wk\") pod \"c4b3939b-c274-47f1-a31f-82245276004f\" (UID: \"c4b3939b-c274-47f1-a31f-82245276004f\") "
Oct 30 23:06:44.051164 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:44.051119    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4b3939b-c274-47f1-a31f-82245276004f-webhook-cert" pod "c4b3939b-c274-47f1-a31f-82245276004f" (UID: "c4b3939b-c274-47f1-a31f-82245276004f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:06:44.051164 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:44.051144    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4b3939b-c274-47f1-a31f-82245276004f-metrics-certs" pod "c4b3939b-c274-47f1-a31f-82245276004f" (UID: "c4b3939b-c274-47f1-a31f-82245276004f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:06:44.051453 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:44.051194    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4b3939b-c274-47f1-a31f-82245276004f-kube-api-access-8g7wk" pod "c4b3939b-c274-47f1-a31f-82245276004f" (UID: "c4b3939b-c274-47f1-a31f-82245276004f"). InnerVolumeSpecName "kube-api-access-8g7wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:06:44.150036 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:44.149955    4889 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4b3939b-c274-47f1-a31f-82245276004f-webhook-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:06:44.150036 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:44.149983    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g7wk\" (UniqueName: \"kubernetes.io/projected/c4b3939b-c274-47f1-a31f-82245276004f-kube-api-access-8g7wk\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:06:44.150036 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:44.149997    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4b3939b-c274-47f1-a31f-82245276004f-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:06:44.600530 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:44.600492    4889 generic.go:356] "Generic (PLEG): container finished" podID="c4b3939b-c274-47f1-a31f-82245276004f" containerID="4ac1905c8dc19bafc5d71f5abd43b7b4bf7894e66676d8e0055c75c9702302f1" exitCode=0
Oct 30 23:06:44.601092 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:44.600581    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8586c445d6-7dkgj"
Oct 30 23:06:44.601092 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:44.600607    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8586c445d6-7dkgj" event={"ID":"c4b3939b-c274-47f1-a31f-82245276004f","Type":"ContainerDied","Data":"4ac1905c8dc19bafc5d71f5abd43b7b4bf7894e66676d8e0055c75c9702302f1"}
Oct 30 23:06:44.601092 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:44.600666    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8586c445d6-7dkgj" event={"ID":"c4b3939b-c274-47f1-a31f-82245276004f","Type":"ContainerDied","Data":"4b6710aa0ef5850fbbe5bb9948c093c70366df47333087dd82b17509e692c74e"}
Oct 30 23:06:44.601092 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:44.600689    4889 scope.go:122] "RemoveContainer" containerID="4ac1905c8dc19bafc5d71f5abd43b7b4bf7894e66676d8e0055c75c9702302f1"
Oct 30 23:06:44.613088 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:44.613068    4889 scope.go:122] "RemoveContainer" containerID="4ac1905c8dc19bafc5d71f5abd43b7b4bf7894e66676d8e0055c75c9702302f1"
Oct 30 23:06:44.613954 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:06:44.613931    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ac1905c8dc19bafc5d71f5abd43b7b4bf7894e66676d8e0055c75c9702302f1\": container with ID starting with 4ac1905c8dc19bafc5d71f5abd43b7b4bf7894e66676d8e0055c75c9702302f1 not found: ID does not exist" containerID="4ac1905c8dc19bafc5d71f5abd43b7b4bf7894e66676d8e0055c75c9702302f1"
Oct 30 23:06:44.614025 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:44.613965    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac1905c8dc19bafc5d71f5abd43b7b4bf7894e66676d8e0055c75c9702302f1"} err="failed to get container status \"4ac1905c8dc19bafc5d71f5abd43b7b4bf7894e66676d8e0055c75c9702302f1\": rpc error: code = NotFound desc = could not find container \"4ac1905c8dc19bafc5d71f5abd43b7b4bf7894e66676d8e0055c75c9702302f1\": container with ID starting with 4ac1905c8dc19bafc5d71f5abd43b7b4bf7894e66676d8e0055c75c9702302f1 not found: ID does not exist"
Oct 30 23:06:44.657414 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:44.657382    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8586c445d6-7dkgj"]
Oct 30 23:06:44.670828 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:44.670801    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8586c445d6-7dkgj"]
Oct 30 23:06:44.775183 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:44.775145    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4b3939b-c274-47f1-a31f-82245276004f" path="/var/lib/kubelet/pods/c4b3939b-c274-47f1-a31f-82245276004f/volumes"
Oct 30 23:06:45.758540 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:45.758504    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:06:45.816928 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:45.816895    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8586c445d6-vh5wm"]
Oct 30 23:06:45.817312 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:45.817288    4889 cpu_manager.go:414] "containerMap: removing container" podUID="c4b3939b-c274-47f1-a31f-82245276004f" containerName="manager"
Oct 30 23:06:45.817312 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:45.817314    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4b3939b-c274-47f1-a31f-82245276004f" containerName="manager"
Oct 30 23:06:45.817438 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:45.817386    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4b3939b-c274-47f1-a31f-82245276004f" containerName="manager"
Oct 30 23:06:45.822029 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:45.822012    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8586c445d6-vh5wm"
Oct 30 23:06:45.839155 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:45.839132    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8586c445d6-vh5wm"]
Oct 30 23:06:45.850593 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:45.850568    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 23:06:45.853687 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:45.853664    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jthrb\""
Oct 30 23:06:45.853829 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:45.853742    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 23:06:45.853902 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:45.853837    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 23:06:45.862638 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:45.862615    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/457b7ef2-52f7-4589-a3a5-eb76e5d6ae17-metrics-certs\") pod \"maas-controller-8586c445d6-vh5wm\" (UID: \"457b7ef2-52f7-4589-a3a5-eb76e5d6ae17\") " pod="opendatahub/maas-controller-8586c445d6-vh5wm"
Oct 30 23:06:45.862766 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:45.862664    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/457b7ef2-52f7-4589-a3a5-eb76e5d6ae17-webhook-cert\") pod \"maas-controller-8586c445d6-vh5wm\" (UID: \"457b7ef2-52f7-4589-a3a5-eb76e5d6ae17\") " pod="opendatahub/maas-controller-8586c445d6-vh5wm"
Oct 30 23:06:45.862840 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:45.862804    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk5n8\" (UniqueName: \"kubernetes.io/projected/457b7ef2-52f7-4589-a3a5-eb76e5d6ae17-kube-api-access-mk5n8\") pod \"maas-controller-8586c445d6-vh5wm\" (UID: \"457b7ef2-52f7-4589-a3a5-eb76e5d6ae17\") " pod="opendatahub/maas-controller-8586c445d6-vh5wm"
Oct 30 23:06:45.964067 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:45.964035    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/457b7ef2-52f7-4589-a3a5-eb76e5d6ae17-webhook-cert\") pod \"maas-controller-8586c445d6-vh5wm\" (UID: \"457b7ef2-52f7-4589-a3a5-eb76e5d6ae17\") " pod="opendatahub/maas-controller-8586c445d6-vh5wm"
Oct 30 23:06:45.964269 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:45.964084    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk5n8\" (UniqueName: \"kubernetes.io/projected/457b7ef2-52f7-4589-a3a5-eb76e5d6ae17-kube-api-access-mk5n8\") pod \"maas-controller-8586c445d6-vh5wm\" (UID: \"457b7ef2-52f7-4589-a3a5-eb76e5d6ae17\") " pod="opendatahub/maas-controller-8586c445d6-vh5wm"
Oct 30 23:06:45.964269 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:45.964165    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/457b7ef2-52f7-4589-a3a5-eb76e5d6ae17-metrics-certs\") pod \"maas-controller-8586c445d6-vh5wm\" (UID: \"457b7ef2-52f7-4589-a3a5-eb76e5d6ae17\") " pod="opendatahub/maas-controller-8586c445d6-vh5wm"
Oct 30 23:06:45.966695 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:45.966670    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/457b7ef2-52f7-4589-a3a5-eb76e5d6ae17-webhook-cert\") pod \"maas-controller-8586c445d6-vh5wm\" (UID: \"457b7ef2-52f7-4589-a3a5-eb76e5d6ae17\") " pod="opendatahub/maas-controller-8586c445d6-vh5wm"
Oct 30 23:06:45.966811 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:45.966723    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/457b7ef2-52f7-4589-a3a5-eb76e5d6ae17-metrics-certs\") pod \"maas-controller-8586c445d6-vh5wm\" (UID: \"457b7ef2-52f7-4589-a3a5-eb76e5d6ae17\") " pod="opendatahub/maas-controller-8586c445d6-vh5wm"
Oct 30 23:06:45.985597 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:45.985570    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk5n8\" (UniqueName: \"kubernetes.io/projected/457b7ef2-52f7-4589-a3a5-eb76e5d6ae17-kube-api-access-mk5n8\") pod \"maas-controller-8586c445d6-vh5wm\" (UID: \"457b7ef2-52f7-4589-a3a5-eb76e5d6ae17\") " pod="opendatahub/maas-controller-8586c445d6-vh5wm"
Oct 30 23:06:46.134454 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:46.134364    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8586c445d6-vh5wm"
Oct 30 23:06:46.432968 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:46.432942    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8586c445d6-vh5wm"]
Oct 30 23:06:46.434748 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:06:46.434720    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457b7ef2_52f7_4589_a3a5_eb76e5d6ae17.slice/crio-e86e4bc3ae3f99dd18e92ee478cdcb2d1fe0871f17be981f0686ce4690af880f WatchSource:0}: Error finding container e86e4bc3ae3f99dd18e92ee478cdcb2d1fe0871f17be981f0686ce4690af880f: Status 404 returned error can't find the container with id e86e4bc3ae3f99dd18e92ee478cdcb2d1fe0871f17be981f0686ce4690af880f
Oct 30 23:06:46.611757 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:46.611721    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8586c445d6-vh5wm" event={"ID":"457b7ef2-52f7-4589-a3a5-eb76e5d6ae17","Type":"ContainerStarted","Data":"e86e4bc3ae3f99dd18e92ee478cdcb2d1fe0871f17be981f0686ce4690af880f"}
Oct 30 23:06:47.618408 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:47.618373    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8586c445d6-vh5wm" event={"ID":"457b7ef2-52f7-4589-a3a5-eb76e5d6ae17","Type":"ContainerStarted","Data":"3df7f6dc6fde8dbb3bcf84c94e65be3a42a0c7fd18610001951b5a34c8624a9a"}
Oct 30 23:06:47.618923 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:47.618458    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8586c445d6-vh5wm"
Oct 30 23:06:47.665823 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:47.665768    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-8586c445d6-vh5wm" podStartSLOduration=2.311322954 podStartE2EDuration="2.665752804s" totalImagesPullingTime="354.42985ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:06:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:47.664190189 +0000 UTC m=+3593.020556504" watchObservedRunningTime="2026-10-30 23:06:47.665752804 +0000 UTC m=+3593.022119158"
Oct 30 23:06:52.072748 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:52.072715    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:06:55.808344 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:55.808312    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 23:06:55.816734 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:55.816707    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 23:06:58.625877 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:06:58.625845    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8586c445d6-vh5wm"
Oct 30 23:07:05.763400 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:05.763354    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:10.302215 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:07:10.302175    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:07:40.302688 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:07:40.302605    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:07:44.384242 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.384198    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"]
Oct 30 23:07:44.388022 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.387999    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.397941 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.397911    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"]
Oct 30 23:07:44.401351 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.401329    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.412157 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.412130    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"]
Oct 30 23:07:44.420363 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.420334    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17ab4abd-dafe-4275-8740-e71ac3d3fd25-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jp9lg\" (UID: \"17ab4abd-dafe-4275-8740-e71ac3d3fd25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.420513 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.420483    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ab4abd-dafe-4275-8740-e71ac3d3fd25-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jp9lg\" (UID: \"17ab4abd-dafe-4275-8740-e71ac3d3fd25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.420621 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.420530    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtktl\" (UniqueName: \"kubernetes.io/projected/17ab4abd-dafe-4275-8740-e71ac3d3fd25-kube-api-access-jtktl\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jp9lg\" (UID: \"17ab4abd-dafe-4275-8740-e71ac3d3fd25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.420621 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.420559    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ab4abd-dafe-4275-8740-e71ac3d3fd25-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jp9lg\" (UID: \"17ab4abd-dafe-4275-8740-e71ac3d3fd25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.420756 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.420645    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ab4abd-dafe-4275-8740-e71ac3d3fd25-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jp9lg\" (UID: \"17ab4abd-dafe-4275-8740-e71ac3d3fd25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.420756 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.420712    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17ab4abd-dafe-4275-8740-e71ac3d3fd25-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jp9lg\" (UID: \"17ab4abd-dafe-4275-8740-e71ac3d3fd25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.420756 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.420741    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17ab4abd-dafe-4275-8740-e71ac3d3fd25-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jp9lg\" (UID: \"17ab4abd-dafe-4275-8740-e71ac3d3fd25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.420946 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.420773    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17ab4abd-dafe-4275-8740-e71ac3d3fd25-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jp9lg\" (UID: \"17ab4abd-dafe-4275-8740-e71ac3d3fd25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.420946 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.420898    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17ab4abd-dafe-4275-8740-e71ac3d3fd25-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jp9lg\" (UID: \"17ab4abd-dafe-4275-8740-e71ac3d3fd25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.434739 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.434712    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"]
Oct 30 23:07:44.521484 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.521443    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17ab4abd-dafe-4275-8740-e71ac3d3fd25-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jp9lg\" (UID: \"17ab4abd-dafe-4275-8740-e71ac3d3fd25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.521484 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.521483    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500a9955-0466-4536-b274-9c669e5ffffa-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7qjgv\" (UID: \"500a9955-0466-4536-b274-9c669e5ffffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.521699 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.521505    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500a9955-0466-4536-b274-9c669e5ffffa-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7qjgv\" (UID: \"500a9955-0466-4536-b274-9c669e5ffffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.521699 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.521587    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500a9955-0466-4536-b274-9c669e5ffffa-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7qjgv\" (UID: \"500a9955-0466-4536-b274-9c669e5ffffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.521699 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.521640    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17ab4abd-dafe-4275-8740-e71ac3d3fd25-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jp9lg\" (UID: \"17ab4abd-dafe-4275-8740-e71ac3d3fd25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.521837 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.521706    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500a9955-0466-4536-b274-9c669e5ffffa-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7qjgv\" (UID: \"500a9955-0466-4536-b274-9c669e5ffffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.521837 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.521745    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500a9955-0466-4536-b274-9c669e5ffffa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7qjgv\" (UID: \"500a9955-0466-4536-b274-9c669e5ffffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.521837 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.521773    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ab4abd-dafe-4275-8740-e71ac3d3fd25-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jp9lg\" (UID: \"17ab4abd-dafe-4275-8740-e71ac3d3fd25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.521837 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.521793    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500a9955-0466-4536-b274-9c669e5ffffa-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7qjgv\" (UID: \"500a9955-0466-4536-b274-9c669e5ffffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.522071 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.521907    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtktl\" (UniqueName: \"kubernetes.io/projected/17ab4abd-dafe-4275-8740-e71ac3d3fd25-kube-api-access-jtktl\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jp9lg\" (UID: \"17ab4abd-dafe-4275-8740-e71ac3d3fd25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.522071 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.521941    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ab4abd-dafe-4275-8740-e71ac3d3fd25-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jp9lg\" (UID: \"17ab4abd-dafe-4275-8740-e71ac3d3fd25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.522071 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.521969    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ab4abd-dafe-4275-8740-e71ac3d3fd25-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jp9lg\" (UID: \"17ab4abd-dafe-4275-8740-e71ac3d3fd25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.522071 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.522003    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500a9955-0466-4536-b274-9c669e5ffffa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7qjgv\" (UID: \"500a9955-0466-4536-b274-9c669e5ffffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.522071 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.522032    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500a9955-0466-4536-b274-9c669e5ffffa-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7qjgv\" (UID: \"500a9955-0466-4536-b274-9c669e5ffffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.522071 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.522057    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17ab4abd-dafe-4275-8740-e71ac3d3fd25-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jp9lg\" (UID: \"17ab4abd-dafe-4275-8740-e71ac3d3fd25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.522385 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.522112    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17ab4abd-dafe-4275-8740-e71ac3d3fd25-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jp9lg\" (UID: \"17ab4abd-dafe-4275-8740-e71ac3d3fd25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.522385 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.522143    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ab4abd-dafe-4275-8740-e71ac3d3fd25-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jp9lg\" (UID: \"17ab4abd-dafe-4275-8740-e71ac3d3fd25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.522385 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.522150    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17ab4abd-dafe-4275-8740-e71ac3d3fd25-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jp9lg\" (UID: \"17ab4abd-dafe-4275-8740-e71ac3d3fd25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.522385 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.522209    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5w49\" (UniqueName: \"kubernetes.io/projected/500a9955-0466-4536-b274-9c669e5ffffa-kube-api-access-p5w49\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7qjgv\" (UID: \"500a9955-0466-4536-b274-9c669e5ffffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.522385 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.522297    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ab4abd-dafe-4275-8740-e71ac3d3fd25-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jp9lg\" (UID: \"17ab4abd-dafe-4275-8740-e71ac3d3fd25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.522385 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.522298    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17ab4abd-dafe-4275-8740-e71ac3d3fd25-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jp9lg\" (UID: \"17ab4abd-dafe-4275-8740-e71ac3d3fd25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.522780 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.522757    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ab4abd-dafe-4275-8740-e71ac3d3fd25-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jp9lg\" (UID: \"17ab4abd-dafe-4275-8740-e71ac3d3fd25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.522903 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.522887    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17ab4abd-dafe-4275-8740-e71ac3d3fd25-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jp9lg\" (UID: \"17ab4abd-dafe-4275-8740-e71ac3d3fd25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.524673 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.524648    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17ab4abd-dafe-4275-8740-e71ac3d3fd25-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jp9lg\" (UID: \"17ab4abd-dafe-4275-8740-e71ac3d3fd25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.524784 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.524684    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17ab4abd-dafe-4275-8740-e71ac3d3fd25-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jp9lg\" (UID: \"17ab4abd-dafe-4275-8740-e71ac3d3fd25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.561295 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.561267    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17ab4abd-dafe-4275-8740-e71ac3d3fd25-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jp9lg\" (UID: \"17ab4abd-dafe-4275-8740-e71ac3d3fd25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.561482 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.561463    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtktl\" (UniqueName: \"kubernetes.io/projected/17ab4abd-dafe-4275-8740-e71ac3d3fd25-kube-api-access-jtktl\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jp9lg\" (UID: \"17ab4abd-dafe-4275-8740-e71ac3d3fd25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.623214 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.623176    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500a9955-0466-4536-b274-9c669e5ffffa-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7qjgv\" (UID: \"500a9955-0466-4536-b274-9c669e5ffffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.623407 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.623219    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500a9955-0466-4536-b274-9c669e5ffffa-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7qjgv\" (UID: \"500a9955-0466-4536-b274-9c669e5ffffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.623407 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.623281    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500a9955-0466-4536-b274-9c669e5ffffa-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7qjgv\" (UID: \"500a9955-0466-4536-b274-9c669e5ffffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.623407 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.623314    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500a9955-0466-4536-b274-9c669e5ffffa-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7qjgv\" (UID: \"500a9955-0466-4536-b274-9c669e5ffffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.623407 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.623360    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500a9955-0466-4536-b274-9c669e5ffffa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7qjgv\" (UID: \"500a9955-0466-4536-b274-9c669e5ffffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.623407 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.623380    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500a9955-0466-4536-b274-9c669e5ffffa-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7qjgv\" (UID: \"500a9955-0466-4536-b274-9c669e5ffffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.623664 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.623412    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500a9955-0466-4536-b274-9c669e5ffffa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7qjgv\" (UID: \"500a9955-0466-4536-b274-9c669e5ffffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.623664 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.623440    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500a9955-0466-4536-b274-9c669e5ffffa-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7qjgv\" (UID: \"500a9955-0466-4536-b274-9c669e5ffffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.623664 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.623470    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5w49\" (UniqueName: \"kubernetes.io/projected/500a9955-0466-4536-b274-9c669e5ffffa-kube-api-access-p5w49\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7qjgv\" (UID: \"500a9955-0466-4536-b274-9c669e5ffffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.623664 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.623634    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500a9955-0466-4536-b274-9c669e5ffffa-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7qjgv\" (UID: \"500a9955-0466-4536-b274-9c669e5ffffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.623901 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.623688    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500a9955-0466-4536-b274-9c669e5ffffa-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7qjgv\" (UID: \"500a9955-0466-4536-b274-9c669e5ffffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.623901 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.623808    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500a9955-0466-4536-b274-9c669e5ffffa-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7qjgv\" (UID: \"500a9955-0466-4536-b274-9c669e5ffffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.624006 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.623920    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500a9955-0466-4536-b274-9c669e5ffffa-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7qjgv\" (UID: \"500a9955-0466-4536-b274-9c669e5ffffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.624085 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.624066    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500a9955-0466-4536-b274-9c669e5ffffa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7qjgv\" (UID: \"500a9955-0466-4536-b274-9c669e5ffffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.626165 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.626145    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500a9955-0466-4536-b274-9c669e5ffffa-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7qjgv\" (UID: \"500a9955-0466-4536-b274-9c669e5ffffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.626305 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.626288    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500a9955-0466-4536-b274-9c669e5ffffa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7qjgv\" (UID: \"500a9955-0466-4536-b274-9c669e5ffffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.649283 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.649196    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500a9955-0466-4536-b274-9c669e5ffffa-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7qjgv\" (UID: \"500a9955-0466-4536-b274-9c669e5ffffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.649483 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.649414    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5w49\" (UniqueName: \"kubernetes.io/projected/500a9955-0466-4536-b274-9c669e5ffffa-kube-api-access-p5w49\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7qjgv\" (UID: \"500a9955-0466-4536-b274-9c669e5ffffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:44.702568 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.702534    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:44.721976 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:44.721950    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:45.023137 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:45.023107    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"]
Oct 30 23:07:45.023137 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:07:45.023127    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ab4abd_dafe_4275_8740_e71ac3d3fd25.slice/crio-6716ee878c2a42efd88a3d7e3ff8f853cdd1a23b562c262f880e0e60a3746fa5 WatchSource:0}: Error finding container 6716ee878c2a42efd88a3d7e3ff8f853cdd1a23b562c262f880e0e60a3746fa5: Status 404 returned error can't find the container with id 6716ee878c2a42efd88a3d7e3ff8f853cdd1a23b562c262f880e0e60a3746fa5
Oct 30 23:07:45.025505 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:45.025477    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:07:45.025569 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:45.025540    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:07:45.025632 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:45.025569    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:07:45.063761 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:45.063732    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"]
Oct 30 23:07:45.064881 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:07:45.064850    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500a9955_0466_4536_b274_9c669e5ffffa.slice/crio-00f8a0144d6950dd382fbd274db4a139039dd77dc5e4499bfed7642364b7aae9 WatchSource:0}: Error finding container 00f8a0144d6950dd382fbd274db4a139039dd77dc5e4499bfed7642364b7aae9: Status 404 returned error can't find the container with id 00f8a0144d6950dd382fbd274db4a139039dd77dc5e4499bfed7642364b7aae9
Oct 30 23:07:45.067255 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:45.067206    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:07:45.067350 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:45.067288    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:07:45.067350 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:45.067325    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:07:45.893276 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:45.893218    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg" event={"ID":"17ab4abd-dafe-4275-8740-e71ac3d3fd25","Type":"ContainerStarted","Data":"2bea543674c8fe8aabb1d0fd987b5162d3835275296d33cbf4bd5268afa74f58"}
Oct 30 23:07:45.893751 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:45.893294    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg" event={"ID":"17ab4abd-dafe-4275-8740-e71ac3d3fd25","Type":"ContainerStarted","Data":"6716ee878c2a42efd88a3d7e3ff8f853cdd1a23b562c262f880e0e60a3746fa5"}
Oct 30 23:07:45.894726 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:45.894706    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv" event={"ID":"500a9955-0466-4536-b274-9c669e5ffffa","Type":"ContainerStarted","Data":"2ecb9158354074b44acbb8b0624b391089788ecd33403aae9aa33be78fc46609"}
Oct 30 23:07:45.894802 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:45.894743    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv" event={"ID":"500a9955-0466-4536-b274-9c669e5ffffa","Type":"ContainerStarted","Data":"00f8a0144d6950dd382fbd274db4a139039dd77dc5e4499bfed7642364b7aae9"}
Oct 30 23:07:45.970062 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:45.970008    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg" podStartSLOduration=1.969992429 podStartE2EDuration="1.969992429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:07:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:07:45.967672813 +0000 UTC m=+3651.324039130" watchObservedRunningTime="2026-10-30 23:07:45.969992429 +0000 UTC m=+3651.326358743"
Oct 30 23:07:46.703433 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:46.703387    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:46.705098 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:46.705075    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-jp9lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 23:07:46.705217 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:46.705123    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg" podUID="17ab4abd-dafe-4275-8740-e71ac3d3fd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 23:07:46.733249 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:46.731855    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:46.733249 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:46.732013    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7qjgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body=
Oct 30 23:07:46.733249 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:46.732060    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv" podUID="500a9955-0466-4536-b274-9c669e5ffffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused"
Oct 30 23:07:47.703041 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:47.702988    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-jp9lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 23:07:47.703489 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:47.703054    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg" podUID="17ab4abd-dafe-4275-8740-e71ac3d3fd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 23:07:47.725887 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:47.725846    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7qjgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body=
Oct 30 23:07:47.726204 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:47.726172    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv" podUID="500a9955-0466-4536-b274-9c669e5ffffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused"
Oct 30 23:07:48.703547 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:48.703502    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-jp9lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 23:07:48.703978 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:48.703561    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg" podUID="17ab4abd-dafe-4275-8740-e71ac3d3fd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 23:07:48.726250 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:48.726167    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7qjgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body=
Oct 30 23:07:48.726475 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:48.726260    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv" podUID="500a9955-0466-4536-b274-9c669e5ffffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused"
Oct 30 23:07:49.705419 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:49.705356    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-jp9lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 23:07:49.705986 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:49.705446    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg" podUID="17ab4abd-dafe-4275-8740-e71ac3d3fd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 23:07:49.737261 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:49.734140    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7qjgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body=
Oct 30 23:07:49.737261 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:49.734204    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv" podUID="500a9955-0466-4536-b274-9c669e5ffffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused"
Oct 30 23:07:50.703763 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:50.703720    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-jp9lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 23:07:50.704069 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:50.704036    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg" podUID="17ab4abd-dafe-4275-8740-e71ac3d3fd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 23:07:50.728701 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:50.728659    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7qjgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body=
Oct 30 23:07:50.729380 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:50.729348    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv" podUID="500a9955-0466-4536-b274-9c669e5ffffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused"
Oct 30 23:07:51.703268 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:51.703214    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-jp9lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 23:07:51.703515 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:51.703287    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg" podUID="17ab4abd-dafe-4275-8740-e71ac3d3fd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 23:07:51.722888 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:51.722843    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7qjgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body=
Oct 30 23:07:51.723077 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:51.722892    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv" podUID="500a9955-0466-4536-b274-9c669e5ffffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused"
Oct 30 23:07:52.187623 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:52.187581    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:52.702946 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:52.702899    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-jp9lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 23:07:52.703276 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:52.702961    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg" podUID="17ab4abd-dafe-4275-8740-e71ac3d3fd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 23:07:52.722408 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:52.722368    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7qjgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body=
Oct 30 23:07:52.722569 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:52.722418    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv" podUID="500a9955-0466-4536-b274-9c669e5ffffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused"
Oct 30 23:07:54.723336 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:54.723294    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7qjgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:07:54.723960 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:54.723919    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv" podUID="500a9955-0466-4536-b274-9c669e5ffffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:07:54.746450 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:54.746398    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-jp9lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:07:54.746673 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:54.746514    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg" podUID="17ab4abd-dafe-4275-8740-e71ac3d3fd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:07:55.765441 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:55.765399    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7qjgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:07:55.766193 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:55.766050    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv" podUID="500a9955-0466-4536-b274-9c669e5ffffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:07:55.787353 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:55.787326    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-jp9lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:07:55.787492 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:55.787377    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg" podUID="17ab4abd-dafe-4275-8740-e71ac3d3fd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:07:55.952068 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:55.952033    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:55.953159 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:55.953134    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:55.996210 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:55.996184    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg"
Oct 30 23:07:56.044727 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:56.044618    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv" podStartSLOduration=12.04459408 podStartE2EDuration="12.04459408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:07:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:07:46.036881725 +0000 UTC m=+3651.393248041" watchObservedRunningTime="2026-10-30 23:07:56.04459408 +0000 UTC m=+3661.400960394"
Oct 30 23:07:56.261019 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:56.260986    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:56.261440 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:56.261412    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:07:56.338766 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:07:56.338682    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv"
Oct 30 23:08:10.301175 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:08:10.301138    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:08:40.301079 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:08:40.301037    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:08:52.252576 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:08:52.252527    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:08:55.782785 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:08:55.782746    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:08:58.520503 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:08:58.520438    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:09:10.301335 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:09:10.301218    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:09:14.113484 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:09:14.113443    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:09:40.300672 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:09:40.300621    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:09:47.044281 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:09:47.044247    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51c10141-5pjp6"]
Oct 30 23:09:47.057430 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:09:47.057400    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51c10141-5pjp6"]
Oct 30 23:09:48.772500 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:09:48.772458    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d893ba75-1d6a-404b-be7d-8b78ca3e6c62" path="/var/lib/kubelet/pods/d893ba75-1d6a-404b-be7d-8b78ca3e6c62/volumes"
Oct 30 23:09:52.435775 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:09:52.435733    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:10:10.300386 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:10:10.300345    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:10:13.371731 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:10:13.371692    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:10:40.300809 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:10:40.300718    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:10:46.028652 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:10:46.028605    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c55f8ff-vqddm"]
Oct 30 23:10:46.041988 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:10:46.041951    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c55f8ff-vqddm"]
Oct 30 23:10:46.772712 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:10:46.772683    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b6b5ac7-8fa1-4443-840e-4b75885f0bad" path="/var/lib/kubelet/pods/2b6b5ac7-8fa1-4443-840e-4b75885f0bad/volumes"
Oct 30 23:10:52.536376 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:10:52.536340    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:10:57.749908 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:10:57.749878    4889 scope.go:122] "RemoveContainer" containerID="027bce2ba27c46152de8cbda56f7472ce1f6986de2994af09c70284933c1ac7e"
Oct 30 23:10:57.762824 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:10:57.762800    4889 scope.go:122] "RemoveContainer" containerID="678617f0ad52ecb5aa16943d80d253236ba29e2b6e90c27ab57730a39782fd38"
Oct 30 23:11:10.299557 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:11:10.299515    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:11:40.301347 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:11:40.301302    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:11:52.656853 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:52.656799    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:11:53.572940 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:53.572893    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:11:53.903122 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:53.903011    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"]
Oct 30 23:11:53.906890 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:53.906863    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:53.926466 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:53.926433    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-2cad7e-openshift-default-dockercfg-5hb8k\""
Oct 30 23:11:53.943213 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:53.943176    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"]
Oct 30 23:11:54.022567 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.022507    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/833e155d-05c6-4615-acad-bda7aa964a5e-istio-token\") pod \"e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") " pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.023063 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.022909    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-istio-envoy\") pod \"e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") " pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.025031 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.024825    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:11:54.025448 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.025045    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/833e155d-05c6-4615-acad-bda7aa964a5e-istio-podinfo\") pod \"e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") " pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.025448 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.025090    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msstj\" (UniqueName: \"kubernetes.io/projected/833e155d-05c6-4615-acad-bda7aa964a5e-kube-api-access-msstj\") pod \"e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") " pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.025448 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.025165    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-workload-socket\") pod \"e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") " pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.025448 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.025198    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-workload-certs\") pod \"e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") " pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.025448 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.025257    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-credential-socket\") pod \"e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") " pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.026413 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.026382    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-istio-data\") pod \"e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") " pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.026620 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.026604    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/833e155d-05c6-4615-acad-bda7aa964a5e-istiod-ca-cert\") pod \"e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") " pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.127522 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.127478    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-istio-envoy\") pod \"e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") " pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.127760 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.127542    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/833e155d-05c6-4615-acad-bda7aa964a5e-istio-podinfo\") pod \"e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") " pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.127760 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.127564    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msstj\" (UniqueName: \"kubernetes.io/projected/833e155d-05c6-4615-acad-bda7aa964a5e-kube-api-access-msstj\") pod \"e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") " pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.127760 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.127595    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-workload-socket\") pod \"e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") " pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.127760 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.127617    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-workload-certs\") pod \"e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") " pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.127760 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.127634    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-credential-socket\") pod \"e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") " pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.127760 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.127653    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-istio-data\") pod \"e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") " pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.128146 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.127820    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/833e155d-05c6-4615-acad-bda7aa964a5e-istiod-ca-cert\") pod \"e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") " pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.128146 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.127916    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/833e155d-05c6-4615-acad-bda7aa964a5e-istio-token\") pod \"e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") " pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.128146 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.128088    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-workload-socket\") pod \"e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") " pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.128146 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.128109    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-credential-socket\") pod \"e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") " pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.128457 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.128390    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-istio-data\") pod \"e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") " pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.128611 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.128589    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-workload-certs\") pod \"e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") " pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.128766 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.128737    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/833e155d-05c6-4615-acad-bda7aa964a5e-istiod-ca-cert\") pod \"e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") " pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.130266 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.130247    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-istio-envoy\") pod \"e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") " pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.130511 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.130495    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/833e155d-05c6-4615-acad-bda7aa964a5e-istio-podinfo\") pod \"e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") " pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.149458 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.149417    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/833e155d-05c6-4615-acad-bda7aa964a5e-istio-token\") pod \"e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") " pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.149641 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.149589    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msstj\" (UniqueName: \"kubernetes.io/projected/833e155d-05c6-4615-acad-bda7aa964a5e-kube-api-access-msstj\") pod \"e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") " pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.223867 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.223767    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:54.244139 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.244109    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:11:54.544522 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.544494    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"]
Oct 30 23:11:54.545077 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:11:54.545055    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod833e155d_05c6_4615_acad_bda7aa964a5e.slice/crio-71cfd31cd40904d208f4b774b41e5364cefb38a171b1f195cd32e5a575d547bc WatchSource:0}: Error finding container 71cfd31cd40904d208f4b774b41e5364cefb38a171b1f195cd32e5a575d547bc: Status 404 returned error can't find the container with id 71cfd31cd40904d208f4b774b41e5364cefb38a171b1f195cd32e5a575d547bc
Oct 30 23:11:54.547346 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.547313    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:11:54.547417 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.547380    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:11:54.547417 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.547408    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:11:54.571323 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:54.571285    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj" event={"ID":"833e155d-05c6-4615-acad-bda7aa964a5e","Type":"ContainerStarted","Data":"71cfd31cd40904d208f4b774b41e5364cefb38a171b1f195cd32e5a575d547bc"}
Oct 30 23:11:55.577094 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:55.577058    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj" event={"ID":"833e155d-05c6-4615-acad-bda7aa964a5e","Type":"ContainerStarted","Data":"6f8f79bea637ae4a91f136d29783a3423e59697f00bbb2dd316d930b76af83b9"}
Oct 30 23:11:55.648882 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:55.648829    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj" podStartSLOduration=2.648814518 podStartE2EDuration="2.648814518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:11:55.634836404 +0000 UTC m=+3900.991202730" watchObservedRunningTime="2026-10-30 23:11:55.648814518 +0000 UTC m=+3901.005180832"
Oct 30 23:11:55.840014 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:55.839929    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 23:11:55.849710 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:55.849678    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 23:11:56.224809 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:56.224712    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:56.229898 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:56.229871    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:56.582374 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:56.582345    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:56.583602 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:56.583580    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:11:57.786247 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:57.786197    4889 scope.go:122] "RemoveContainer" containerID="44c5d951f8b9234d29299624ce02606c10c4d957dc88e407068890b203271af1"
Oct 30 23:11:58.101653 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:58.101566    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:11:58.270292 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:58.270247    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:11:59.448465 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.448424    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2"]
Oct 30 23:11:59.453521 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.453498    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2"
Oct 30 23:11:59.476101 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.476060    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfv7p\" (UniqueName: \"kubernetes.io/projected/92783ce8-6296-418c-90d3-1807a1a8a3a0-kube-api-access-kfv7p\") pod \"maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2\" (UID: \"92783ce8-6296-418c-90d3-1807a1a8a3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2"
Oct 30 23:11:59.476331 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.476259    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92783ce8-6296-418c-90d3-1807a1a8a3a0-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2\" (UID: \"92783ce8-6296-418c-90d3-1807a1a8a3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2"
Oct 30 23:11:59.476409 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.476313    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92783ce8-6296-418c-90d3-1807a1a8a3a0-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2\" (UID: \"92783ce8-6296-418c-90d3-1807a1a8a3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2"
Oct 30 23:11:59.476409 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.476365    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92783ce8-6296-418c-90d3-1807a1a8a3a0-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2\" (UID: \"92783ce8-6296-418c-90d3-1807a1a8a3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2"
Oct 30 23:11:59.484823 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.484739    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-2cad7e-serving-cert\""
Oct 30 23:11:59.487869 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.487837    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2"]
Oct 30 23:11:59.576969 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.576929    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfv7p\" (UniqueName: \"kubernetes.io/projected/92783ce8-6296-418c-90d3-1807a1a8a3a0-kube-api-access-kfv7p\") pod \"maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2\" (UID: \"92783ce8-6296-418c-90d3-1807a1a8a3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2"
Oct 30 23:11:59.577187 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.577024    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92783ce8-6296-418c-90d3-1807a1a8a3a0-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2\" (UID: \"92783ce8-6296-418c-90d3-1807a1a8a3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2"
Oct 30 23:11:59.577187 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.577054    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92783ce8-6296-418c-90d3-1807a1a8a3a0-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2\" (UID: \"92783ce8-6296-418c-90d3-1807a1a8a3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2"
Oct 30 23:11:59.577187 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.577071    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92783ce8-6296-418c-90d3-1807a1a8a3a0-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2\" (UID: \"92783ce8-6296-418c-90d3-1807a1a8a3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2"
Oct 30 23:11:59.577963 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.577878    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92783ce8-6296-418c-90d3-1807a1a8a3a0-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2\" (UID: \"92783ce8-6296-418c-90d3-1807a1a8a3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2"
Oct 30 23:11:59.579835 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.579810    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92783ce8-6296-418c-90d3-1807a1a8a3a0-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2\" (UID: \"92783ce8-6296-418c-90d3-1807a1a8a3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2"
Oct 30 23:11:59.579933 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.579854    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92783ce8-6296-418c-90d3-1807a1a8a3a0-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2\" (UID: \"92783ce8-6296-418c-90d3-1807a1a8a3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2"
Oct 30 23:11:59.599729 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.599698    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:11:59.599729 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.599712    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfv7p\" (UniqueName: \"kubernetes.io/projected/92783ce8-6296-418c-90d3-1807a1a8a3a0-kube-api-access-kfv7p\") pod \"maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2\" (UID: \"92783ce8-6296-418c-90d3-1807a1a8a3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2"
Oct 30 23:11:59.766997 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.766948    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2"
Oct 30 23:11:59.771885 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.771843    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:11:59.858308 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.858271    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws"]
Oct 30 23:11:59.864421 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.862271    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws"
Oct 30 23:11:59.882261 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.880327    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749bf3ff-6eba-4741-b072-ed34fdc62cf6-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws\" (UID: \"749bf3ff-6eba-4741-b072-ed34fdc62cf6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws"
Oct 30 23:11:59.882261 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.880407    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52ng9\" (UniqueName: \"kubernetes.io/projected/749bf3ff-6eba-4741-b072-ed34fdc62cf6-kube-api-access-52ng9\") pod \"payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws\" (UID: \"749bf3ff-6eba-4741-b072-ed34fdc62cf6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws"
Oct 30 23:11:59.882261 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.880767    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-2cad7e\""
Oct 30 23:11:59.882261 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.880931    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-94cc4baf-dockercfg-r468l\""
Oct 30 23:11:59.882261 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.881235    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws"]
Oct 30 23:11:59.924315 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.924286    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4"]
Oct 30 23:11:59.927602 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.927584    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4"
Oct 30 23:11:59.939392 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.939365    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:11:59.946101 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.946074    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4"]
Oct 30 23:11:59.985247 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.984274    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d64rg\" (UniqueName: \"kubernetes.io/projected/f1617c11-0158-43f0-ad92-c658f3264433-kube-api-access-d64rg\") pod \"payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4\" (UID: \"f1617c11-0158-43f0-ad92-c658f3264433\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4"
Oct 30 23:11:59.985247 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.984331    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749bf3ff-6eba-4741-b072-ed34fdc62cf6-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws\" (UID: \"749bf3ff-6eba-4741-b072-ed34fdc62cf6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws"
Oct 30 23:11:59.985247 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.984387    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52ng9\" (UniqueName: \"kubernetes.io/projected/749bf3ff-6eba-4741-b072-ed34fdc62cf6-kube-api-access-52ng9\") pod \"payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws\" (UID: \"749bf3ff-6eba-4741-b072-ed34fdc62cf6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws"
Oct 30 23:11:59.985247 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.984436    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1617c11-0158-43f0-ad92-c658f3264433-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4\" (UID: \"f1617c11-0158-43f0-ad92-c658f3264433\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4"
Oct 30 23:12:00.001571 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:11:59.997134    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749bf3ff-6eba-4741-b072-ed34fdc62cf6-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws\" (UID: \"749bf3ff-6eba-4741-b072-ed34fdc62cf6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws"
Oct 30 23:12:00.011263 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:00.011198    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52ng9\" (UniqueName: \"kubernetes.io/projected/749bf3ff-6eba-4741-b072-ed34fdc62cf6-kube-api-access-52ng9\") pod \"payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws\" (UID: \"749bf3ff-6eba-4741-b072-ed34fdc62cf6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws"
Oct 30 23:12:00.072786 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:00.072754    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2"]
Oct 30 23:12:00.073869 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:12:00.073847    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92783ce8_6296_418c_90d3_1807a1a8a3a0.slice/crio-718ca5905568efd8e32336905dc26d9410d2f791e943fec217ce9a476f18fbad WatchSource:0}: Error finding container 718ca5905568efd8e32336905dc26d9410d2f791e943fec217ce9a476f18fbad: Status 404 returned error can't find the container with id 718ca5905568efd8e32336905dc26d9410d2f791e943fec217ce9a476f18fbad
Oct 30 23:12:00.075303 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:00.075216    4889 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:12:00.085690 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:00.085653    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d64rg\" (UniqueName: \"kubernetes.io/projected/f1617c11-0158-43f0-ad92-c658f3264433-kube-api-access-d64rg\") pod \"payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4\" (UID: \"f1617c11-0158-43f0-ad92-c658f3264433\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4"
Oct 30 23:12:00.085838 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:00.085742    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1617c11-0158-43f0-ad92-c658f3264433-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4\" (UID: \"f1617c11-0158-43f0-ad92-c658f3264433\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4"
Oct 30 23:12:00.086450 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:00.086431    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1617c11-0158-43f0-ad92-c658f3264433-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4\" (UID: \"f1617c11-0158-43f0-ad92-c658f3264433\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4"
Oct 30 23:12:00.109174 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:00.109143    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d64rg\" (UniqueName: \"kubernetes.io/projected/f1617c11-0158-43f0-ad92-c658f3264433-kube-api-access-d64rg\") pod \"payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4\" (UID: \"f1617c11-0158-43f0-ad92-c658f3264433\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4"
Oct 30 23:12:00.177209 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:00.177174    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws"
Oct 30 23:12:00.242347 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:00.242153    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4"
Oct 30 23:12:00.480555 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:00.480530    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws"]
Oct 30 23:12:00.481832 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:12:00.481808    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod749bf3ff_6eba_4741_b072_ed34fdc62cf6.slice/crio-e39a3a23c479dd9332a6e3ef3b5846bc335e633b442fd1404d6c981b2ddeccf1 WatchSource:0}: Error finding container e39a3a23c479dd9332a6e3ef3b5846bc335e633b442fd1404d6c981b2ddeccf1: Status 404 returned error can't find the container with id e39a3a23c479dd9332a6e3ef3b5846bc335e633b442fd1404d6c981b2ddeccf1
Oct 30 23:12:00.556454 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:00.556423    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4"]
Oct 30 23:12:00.557069 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:12:00.557047    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1617c11_0158_43f0_ad92_c658f3264433.slice/crio-a565bdd72eba7fc1758ec8826c94273388aaa5a1024303ff2f2e66dd5d3e3e16 WatchSource:0}: Error finding container a565bdd72eba7fc1758ec8826c94273388aaa5a1024303ff2f2e66dd5d3e3e16: Status 404 returned error can't find the container with id a565bdd72eba7fc1758ec8826c94273388aaa5a1024303ff2f2e66dd5d3e3e16
Oct 30 23:12:00.612858 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:00.612817    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2" event={"ID":"92783ce8-6296-418c-90d3-1807a1a8a3a0","Type":"ContainerStarted","Data":"74b92c83982f233aac784bd57e91338cbbda447a4724decb1033d25a0aa8a8f5"}
Oct 30 23:12:00.613071 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:00.612888    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2" event={"ID":"92783ce8-6296-418c-90d3-1807a1a8a3a0","Type":"ContainerStarted","Data":"718ca5905568efd8e32336905dc26d9410d2f791e943fec217ce9a476f18fbad"}
Oct 30 23:12:00.613071 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:00.613001    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2"
Oct 30 23:12:00.614256 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:00.614193    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4" event={"ID":"f1617c11-0158-43f0-ad92-c658f3264433","Type":"ContainerStarted","Data":"a565bdd72eba7fc1758ec8826c94273388aaa5a1024303ff2f2e66dd5d3e3e16"}
Oct 30 23:12:00.618972 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:00.618935    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws" event={"ID":"749bf3ff-6eba-4741-b072-ed34fdc62cf6","Type":"ContainerStarted","Data":"2694add0ff6119dc384b7669e5254b3d63daae8e56395bb9de4c136cedb0f7e5"}
Oct 30 23:12:00.619100 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:00.618986    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws" event={"ID":"749bf3ff-6eba-4741-b072-ed34fdc62cf6","Type":"ContainerStarted","Data":"e39a3a23c479dd9332a6e3ef3b5846bc335e633b442fd1404d6c981b2ddeccf1"}
Oct 30 23:12:00.619171 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:00.619112    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws"
Oct 30 23:12:00.664929 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:00.662727    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2" podStartSLOduration=1.255038558 podStartE2EDuration="1.662707917s" totalImagesPullingTime="407.669359ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:11:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:12:00.658385832 +0000 UTC m=+3906.014752150" watchObservedRunningTime="2026-10-30 23:12:00.662707917 +0000 UTC m=+3906.019074237"
Oct 30 23:12:00.700810 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:00.700752    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws" podStartSLOduration=1.700739831 podStartE2EDuration="1.700739831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:12:00.700411769 +0000 UTC m=+3906.056778085" watchObservedRunningTime="2026-10-30 23:12:00.700739831 +0000 UTC m=+3906.057106140"
Oct 30 23:12:01.391891 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:01.391855    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:01.625986 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:01.625941    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4" event={"ID":"f1617c11-0158-43f0-ad92-c658f3264433","Type":"ContainerStarted","Data":"37fd75133cddf1482060e960d00c3f1686f7583ef8e833ce5af133de9d5359ba"}
Oct 30 23:12:01.626483 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:01.626282    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4"
Oct 30 23:12:01.668661 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:01.668532    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4" podStartSLOduration=2.6685174529999998 podStartE2EDuration="2.668517453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:12:01.668001179 +0000 UTC m=+3907.024367494" watchObservedRunningTime="2026-10-30 23:12:01.668517453 +0000 UTC m=+3907.024883782"
Oct 30 23:12:03.155985 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:03.155951    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:04.751781 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:04.751748    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:06.470859 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:06.470818    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:06.638120 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:06.638091    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2"
Oct 30 23:12:08.270722 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:08.270680    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:10.022128 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:10.022088    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:10.304368 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:12:10.304284    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:12:11.628944 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:11.628916    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws"
Oct 30 23:12:11.822351 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:11.822311    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:12.355289 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:12.353860    4889 patch_prober.go:29] interesting pod/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": read tcp 10.132.0.2:59974->10.132.1.13:15021: read: connection reset by peer" start-of-body=
Oct 30 23:12:12.355289 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:12.354059    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj" podUID="833e155d-05c6-4615-acad-bda7aa964a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": read tcp 10.132.0.2:59974->10.132.1.13:15021: read: connection reset by peer"
Oct 30 23:12:12.635290 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:12.635174    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4"
Oct 30 23:12:12.688845 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:12.688808    4889 generic.go:356] "Generic (PLEG): container finished" podID="833e155d-05c6-4615-acad-bda7aa964a5e" containerID="6f8f79bea637ae4a91f136d29783a3423e59697f00bbb2dd316d930b76af83b9" exitCode=0
Oct 30 23:12:12.689026 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:12.688882    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj" event={"ID":"833e155d-05c6-4615-acad-bda7aa964a5e","Type":"ContainerDied","Data":"6f8f79bea637ae4a91f136d29783a3423e59697f00bbb2dd316d930b76af83b9"}
Oct 30 23:12:12.689341 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:12.689316    4889 scope.go:122] "RemoveContainer" containerID="6f8f79bea637ae4a91f136d29783a3423e59697f00bbb2dd316d930b76af83b9"
Oct 30 23:12:12.690679 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:12.690649    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:12:12.690779 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:12.690721    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:12:12.690779 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:12.690753    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:12:12.995500 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:12.995462    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:13.535941 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:13.535901    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:13.697150 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:13.697110    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj" event={"ID":"833e155d-05c6-4615-acad-bda7aa964a5e","Type":"ContainerStarted","Data":"bcf09d81425c21393724b2f06c05529a96d0b4e73191af1ab990e22be3b6fef7"}
Oct 30 23:12:13.741080 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:13.740584    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:13.936314 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:13.935982    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:14.233802 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:14.232494    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:12:14.233802 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:14.232914    4889 patch_prober.go:29] interesting pod/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body=
Oct 30 23:12:14.233802 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:14.232976    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj" podUID="833e155d-05c6-4615-acad-bda7aa964a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused"
Oct 30 23:12:15.234575 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.234531    4889 patch_prober.go:29] interesting pod/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body=
Oct 30 23:12:15.235179 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.235145    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj" podUID="833e155d-05c6-4615-acad-bda7aa964a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused"
Oct 30 23:12:15.301136 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.301103    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"]
Oct 30 23:12:15.305355 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.305318    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"
Oct 30 23:12:15.342580 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.342541    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-2cad7e\"/\"kube-root-ca.crt\""
Oct 30 23:12:15.343816 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.343783    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"]
Oct 30 23:12:15.344816 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.344793    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-2cad7e\"/\"openshift-service-ca.crt\""
Oct 30 23:12:15.369454 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.369412    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-tmp-dir\") pod \"simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr\" (UID: \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\") " pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"
Oct 30 23:12:15.369783 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.369752    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-dshm\") pod \"simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr\" (UID: \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\") " pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"
Oct 30 23:12:15.369943 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.369799    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-model-cache\") pod \"simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr\" (UID: \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\") " pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"
Oct 30 23:12:15.369943 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.369895    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-home\") pod \"simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr\" (UID: \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\") " pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"
Oct 30 23:12:15.370075 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.369953    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqljj\" (UniqueName: \"kubernetes.io/projected/e1c4d22b-f1d2-4609-992a-fca963ed46c5-kube-api-access-lqljj\") pod \"simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr\" (UID: \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\") " pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"
Oct 30 23:12:15.370075 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.370041    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1c4d22b-f1d2-4609-992a-fca963ed46c5-tls-certs\") pod \"simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr\" (UID: \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\") " pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"
Oct 30 23:12:15.370713 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.370682    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:15.381131 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.381097    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-2cad7e\"/\"default-dockercfg-lnj68\""
Oct 30 23:12:15.381280 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.381149    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-2cad7e\"/\"simulator-single-main-2cad7e-kserve-self-signed-certs\""
Oct 30 23:12:15.471017 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.470981    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1c4d22b-f1d2-4609-992a-fca963ed46c5-tls-certs\") pod \"simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr\" (UID: \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\") " pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"
Oct 30 23:12:15.471281 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.471067    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-tmp-dir\") pod \"simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr\" (UID: \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\") " pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"
Oct 30 23:12:15.471281 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.471121    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-dshm\") pod \"simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr\" (UID: \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\") " pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"
Oct 30 23:12:15.471281 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.471146    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-model-cache\") pod \"simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr\" (UID: \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\") " pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"
Oct 30 23:12:15.471281 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.471183    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-home\") pod \"simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr\" (UID: \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\") " pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"
Oct 30 23:12:15.471281 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.471261    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqljj\" (UniqueName: \"kubernetes.io/projected/e1c4d22b-f1d2-4609-992a-fca963ed46c5-kube-api-access-lqljj\") pod \"simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr\" (UID: \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\") " pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"
Oct 30 23:12:15.471636 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.471612    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-tmp-dir\") pod \"simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr\" (UID: \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\") " pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"
Oct 30 23:12:15.471691 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.471665    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-model-cache\") pod \"simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr\" (UID: \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\") " pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"
Oct 30 23:12:15.471731 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.471710    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-home\") pod \"simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr\" (UID: \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\") " pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"
Oct 30 23:12:15.473786 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.473761    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-dshm\") pod \"simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr\" (UID: \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\") " pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"
Oct 30 23:12:15.473959 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.473939    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1c4d22b-f1d2-4609-992a-fca963ed46c5-tls-certs\") pod \"simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr\" (UID: \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\") " pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"
Oct 30 23:12:15.497829 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.497794    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqljj\" (UniqueName: \"kubernetes.io/projected/e1c4d22b-f1d2-4609-992a-fca963ed46c5-kube-api-access-lqljj\") pod \"simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr\" (UID: \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\") " pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"
Oct 30 23:12:15.578849 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.578805    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:15.650239 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.650188    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"
Oct 30 23:12:15.958306 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:15.957995    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"]
Oct 30 23:12:15.967337 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:12:15.967294    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1c4d22b_f1d2_4609_992a_fca963ed46c5.slice/crio-7c6a821d0f4d098ac139e0b86c799e70e117aa91bf5b2fd645f73cee57299b9b WatchSource:0}: Error finding container 7c6a821d0f4d098ac139e0b86c799e70e117aa91bf5b2fd645f73cee57299b9b: Status 404 returned error can't find the container with id 7c6a821d0f4d098ac139e0b86c799e70e117aa91bf5b2fd645f73cee57299b9b
Oct 30 23:12:16.229751 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:16.229654    4889 patch_prober.go:29] interesting pod/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body=
Oct 30 23:12:16.230016 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:16.229987    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj" podUID="833e155d-05c6-4615-acad-bda7aa964a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused"
Oct 30 23:12:16.742380 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:16.742340    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr" event={"ID":"e1c4d22b-f1d2-4609-992a-fca963ed46c5","Type":"ContainerStarted","Data":"08eaaeb29ff10cac66287ebf3eb30815ff2a3caa066bfe54bdebc5a73c6a5464"}
Oct 30 23:12:16.742837 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:16.742389    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr" event={"ID":"e1c4d22b-f1d2-4609-992a-fca963ed46c5","Type":"ContainerStarted","Data":"7c6a821d0f4d098ac139e0b86c799e70e117aa91bf5b2fd645f73cee57299b9b"}
Oct 30 23:12:16.785119 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:16.785067    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr" podStartSLOduration=1.785052549 podStartE2EDuration="1.785052549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:12:16.784502239 +0000 UTC m=+3922.140868555" watchObservedRunningTime="2026-10-30 23:12:16.785052549 +0000 UTC m=+3922.141418863"
Oct 30 23:12:17.236263 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:17.235545    4889 patch_prober.go:29] interesting pod/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body=
Oct 30 23:12:17.236263 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:17.235628    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj" podUID="833e155d-05c6-4615-acad-bda7aa964a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused"
Oct 30 23:12:17.293112 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:17.293072    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:17.650974 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:17.650933    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"
Oct 30 23:12:18.224717 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:18.224677    4889 patch_prober.go:29] interesting pod/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body=
Oct 30 23:12:18.225391 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:18.225356    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj" podUID="833e155d-05c6-4615-acad-bda7aa964a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused"
Oct 30 23:12:19.054210 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:19.054177    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:19.352042 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:19.351953    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:12:19.352742 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:19.352608    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:12:19.353559 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:19.353533    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:12:20.621906 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:20.621860    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:22.197847 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:22.197814    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:23.910936 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:23.910895    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:25.501525 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:25.501476    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:25.651507 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:25.651469    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"
Oct 30 23:12:25.665249 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:25.665207    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"
Oct 30 23:12:25.799067 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:25.798970    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"
Oct 30 23:12:27.119765 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:27.119721    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:28.813393 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:28.813351    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:30.314647 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:30.314609    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:32.027065 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:32.027027    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:33.812661 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:33.812611    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:35.556387 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:35.556347    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:37.207874 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:37.207830    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:38.835139 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:38.835103    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:40.302533 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:12:40.302494    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:12:40.610282 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:40.610184    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:40.811534 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:40.811495    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:42.410709 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:42.410676    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:44.125298 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:44.125254    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:45.822607 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:45.822562    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:47.538434 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:47.538381    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:49.217410 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:49.217364    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:50.813184 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:50.813145    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:52.409253 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:52.409201    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:52.759491 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:52.759459    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:54.179566 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:54.179531    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:56.396860 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:56.396775    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:57.540474 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:57.540441    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:58.343362 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:58.343321    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:59.235834 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:12:59.235793    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:13:01.103958 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:01.103919    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:13:02.821126 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:02.821080    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:13:03.248403 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:03.248370    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"]
Oct 30 23:13:03.382605 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:03.382562    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:13:04.515241 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:04.515189    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:13:06.225505 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:06.225467    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:13:07.971531 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:07.971488    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:13:08.937258 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:08.937204    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws"]
Oct 30 23:13:08.969066 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:08.969034    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4"]
Oct 30 23:13:09.146720 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:09.146631    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:13:09.324981 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:09.324922    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:13:09.632123 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:09.632016    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:13:10.300845 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:13:10.300797    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:13:11.027286 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:11.027244    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2"]
Oct 30 23:13:11.027573 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:11.027535    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2" podUID="92783ce8-6296-418c-90d3-1807a1a8a3a0" containerName="maas-api" containerID="cri-o://74b92c83982f233aac784bd57e91338cbbda447a4724decb1033d25a0aa8a8f5" gracePeriod=30
Oct 30 23:13:11.189902 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:11.189110    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:13:11.491672 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:11.491641    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2"
Oct 30 23:13:11.663322 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:11.663204    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92783ce8-6296-418c-90d3-1807a1a8a3a0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92783ce8-6296-418c-90d3-1807a1a8a3a0-openshift-service-ca\") pod \"92783ce8-6296-418c-90d3-1807a1a8a3a0\" (UID: \"92783ce8-6296-418c-90d3-1807a1a8a3a0\") "
Oct 30 23:13:11.663518 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:11.663378    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92783ce8-6296-418c-90d3-1807a1a8a3a0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/92783ce8-6296-418c-90d3-1807a1a8a3a0-metrics-certs\") pod \"92783ce8-6296-418c-90d3-1807a1a8a3a0\" (UID: \"92783ce8-6296-418c-90d3-1807a1a8a3a0\") "
Oct 30 23:13:11.663518 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:11.663411    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92783ce8-6296-418c-90d3-1807a1a8a3a0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92783ce8-6296-418c-90d3-1807a1a8a3a0-maas-api-tls\") pod \"92783ce8-6296-418c-90d3-1807a1a8a3a0\" (UID: \"92783ce8-6296-418c-90d3-1807a1a8a3a0\") "
Oct 30 23:13:11.663518 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:11.663451    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92783ce8-6296-418c-90d3-1807a1a8a3a0-kube-api-access-kfv7p\" (UniqueName: \"kubernetes.io/projected/92783ce8-6296-418c-90d3-1807a1a8a3a0-kube-api-access-kfv7p\") pod \"92783ce8-6296-418c-90d3-1807a1a8a3a0\" (UID: \"92783ce8-6296-418c-90d3-1807a1a8a3a0\") "
Oct 30 23:13:11.663694 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:11.663576    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92783ce8-6296-418c-90d3-1807a1a8a3a0-openshift-service-ca" pod "92783ce8-6296-418c-90d3-1807a1a8a3a0" (UID: "92783ce8-6296-418c-90d3-1807a1a8a3a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:13:11.663829 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:11.663811    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92783ce8-6296-418c-90d3-1807a1a8a3a0-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:13:11.665713 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:11.665686    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92783ce8-6296-418c-90d3-1807a1a8a3a0-maas-api-tls" pod "92783ce8-6296-418c-90d3-1807a1a8a3a0" (UID: "92783ce8-6296-418c-90d3-1807a1a8a3a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:13:11.665836 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:11.665770    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92783ce8-6296-418c-90d3-1807a1a8a3a0-kube-api-access-kfv7p" pod "92783ce8-6296-418c-90d3-1807a1a8a3a0" (UID: "92783ce8-6296-418c-90d3-1807a1a8a3a0"). InnerVolumeSpecName "kube-api-access-kfv7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:13:11.665836 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:11.665798    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92783ce8-6296-418c-90d3-1807a1a8a3a0-metrics-certs" pod "92783ce8-6296-418c-90d3-1807a1a8a3a0" (UID: "92783ce8-6296-418c-90d3-1807a1a8a3a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:13:11.764530 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:11.764488    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92783ce8-6296-418c-90d3-1807a1a8a3a0-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:13:11.764530 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:11.764517    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92783ce8-6296-418c-90d3-1807a1a8a3a0-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:13:11.764530 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:11.764526    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfv7p\" (UniqueName: \"kubernetes.io/projected/92783ce8-6296-418c-90d3-1807a1a8a3a0-kube-api-access-kfv7p\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:13:11.846189 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:11.846153    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:13:12.007472 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:12.007434    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:13:12.020771 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:12.020739    4889 generic.go:356] "Generic (PLEG): container finished" podID="92783ce8-6296-418c-90d3-1807a1a8a3a0" containerID="74b92c83982f233aac784bd57e91338cbbda447a4724decb1033d25a0aa8a8f5" exitCode=0
Oct 30 23:13:12.020957 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:12.020800    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2"
Oct 30 23:13:12.020957 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:12.020794    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2" event={"ID":"92783ce8-6296-418c-90d3-1807a1a8a3a0","Type":"ContainerDied","Data":"74b92c83982f233aac784bd57e91338cbbda447a4724decb1033d25a0aa8a8f5"}
Oct 30 23:13:12.021041 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:12.020962    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2" event={"ID":"92783ce8-6296-418c-90d3-1807a1a8a3a0","Type":"ContainerDied","Data":"718ca5905568efd8e32336905dc26d9410d2f791e943fec217ce9a476f18fbad"}
Oct 30 23:13:12.021041 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:12.021001    4889 scope.go:122] "RemoveContainer" containerID="74b92c83982f233aac784bd57e91338cbbda447a4724decb1033d25a0aa8a8f5"
Oct 30 23:13:12.034524 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:12.034349    4889 scope.go:122] "RemoveContainer" containerID="74b92c83982f233aac784bd57e91338cbbda447a4724decb1033d25a0aa8a8f5"
Oct 30 23:13:12.034616 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:13:12.034600    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74b92c83982f233aac784bd57e91338cbbda447a4724decb1033d25a0aa8a8f5\": container with ID starting with 74b92c83982f233aac784bd57e91338cbbda447a4724decb1033d25a0aa8a8f5 not found: ID does not exist" containerID="74b92c83982f233aac784bd57e91338cbbda447a4724decb1033d25a0aa8a8f5"
Oct 30 23:13:12.034663 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:12.034624    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74b92c83982f233aac784bd57e91338cbbda447a4724decb1033d25a0aa8a8f5"} err="failed to get container status \"74b92c83982f233aac784bd57e91338cbbda447a4724decb1033d25a0aa8a8f5\": rpc error: code = NotFound desc = could not find container \"74b92c83982f233aac784bd57e91338cbbda447a4724decb1033d25a0aa8a8f5\": container with ID starting with 74b92c83982f233aac784bd57e91338cbbda447a4724decb1033d25a0aa8a8f5 not found: ID does not exist"
Oct 30 23:13:12.081124 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:12.081091    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2"]
Oct 30 23:13:12.094174 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:12.094146    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cad7e-84f7497b84-t9lg2"]
Oct 30 23:13:12.419456 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:12.419353    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:13:12.772530 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:12.772494    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92783ce8-6296-418c-90d3-1807a1a8a3a0" path="/var/lib/kubelet/pods/92783ce8-6296-418c-90d3-1807a1a8a3a0/volumes"
Oct 30 23:13:13.937861 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:13.937797    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws" podUID="749bf3ff-6eba-4741-b072-ed34fdc62cf6" containerName="payload-processing" containerID="cri-o://2694add0ff6119dc384b7669e5254b3d63daae8e56395bb9de4c136cedb0f7e5" gracePeriod=25
Oct 30 23:13:13.969738 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:13.969699    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4" podUID="f1617c11-0158-43f0-ad92-c658f3264433" containerName="payload-pre-processing" containerID="cri-o://37fd75133cddf1482060e960d00c3f1686f7583ef8e833ce5af133de9d5359ba" gracePeriod=25
Oct 30 23:13:18.269894 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:18.269859    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr" podUID="e1c4d22b-f1d2-4609-992a-fca963ed46c5" containerName="main" containerID="cri-o://08eaaeb29ff10cac66287ebf3eb30815ff2a3caa066bfe54bdebc5a73c6a5464" gracePeriod=45
Oct 30 23:13:18.528409 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:18.528351    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"
Oct 30 23:13:18.632371 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:18.632339    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-tmp-dir\") pod \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\" (UID: \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\") "
Oct 30 23:13:18.632563 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:18.632390    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-dshm\") pod \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\" (UID: \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\") "
Oct 30 23:13:18.632563 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:18.632422    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-home\" (UniqueName: \"kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-home\") pod \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\" (UID: \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\") "
Oct 30 23:13:18.632563 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:18.632477    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1c4d22b-f1d2-4609-992a-fca963ed46c5-kube-api-access-lqljj\" (UniqueName: \"kubernetes.io/projected/e1c4d22b-f1d2-4609-992a-fca963ed46c5-kube-api-access-lqljj\") pod \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\" (UID: \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\") "
Oct 30 23:13:18.632824 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:18.632574    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1c4d22b-f1d2-4609-992a-fca963ed46c5-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1c4d22b-f1d2-4609-992a-fca963ed46c5-tls-certs\") pod \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\" (UID: \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\") "
Oct 30 23:13:18.632824 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:18.632612    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-model-cache\") pod \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\" (UID: \"e1c4d22b-f1d2-4609-992a-fca963ed46c5\") "
Oct 30 23:13:18.632824 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:18.632656    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-tmp-dir" pod "e1c4d22b-f1d2-4609-992a-fca963ed46c5" (UID: "e1c4d22b-f1d2-4609-992a-fca963ed46c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:13:18.632824 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:18.632710    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-home" pod "e1c4d22b-f1d2-4609-992a-fca963ed46c5" (UID: "e1c4d22b-f1d2-4609-992a-fca963ed46c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:13:18.633065 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:18.632908    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-model-cache" pod "e1c4d22b-f1d2-4609-992a-fca963ed46c5" (UID: "e1c4d22b-f1d2-4609-992a-fca963ed46c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:13:18.633065 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:18.632952    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:13:18.633065 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:18.632972    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:13:18.634932 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:18.634905    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-dshm" pod "e1c4d22b-f1d2-4609-992a-fca963ed46c5" (UID: "e1c4d22b-f1d2-4609-992a-fca963ed46c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:13:18.635055 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:18.634986    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1c4d22b-f1d2-4609-992a-fca963ed46c5-tls-certs" pod "e1c4d22b-f1d2-4609-992a-fca963ed46c5" (UID: "e1c4d22b-f1d2-4609-992a-fca963ed46c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:13:18.635055 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:18.635032    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1c4d22b-f1d2-4609-992a-fca963ed46c5-kube-api-access-lqljj" pod "e1c4d22b-f1d2-4609-992a-fca963ed46c5" (UID: "e1c4d22b-f1d2-4609-992a-fca963ed46c5"). InnerVolumeSpecName "kube-api-access-lqljj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:13:18.734460 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:18.734412    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1c4d22b-f1d2-4609-992a-fca963ed46c5-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:13:18.734460 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:18.734455    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:13:18.734460 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:18.734465    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1c4d22b-f1d2-4609-992a-fca963ed46c5-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:13:18.734460 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:18.734474    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqljj\" (UniqueName: \"kubernetes.io/projected/e1c4d22b-f1d2-4609-992a-fca963ed46c5-kube-api-access-lqljj\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:13:18.825843 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:18.825519    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:13:19.007763 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:19.007724    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:13:19.067307 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:19.061883    4889 generic.go:356] "Generic (PLEG): container finished" podID="e1c4d22b-f1d2-4609-992a-fca963ed46c5" containerID="08eaaeb29ff10cac66287ebf3eb30815ff2a3caa066bfe54bdebc5a73c6a5464" exitCode=0
Oct 30 23:13:19.067307 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:19.061966    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"
Oct 30 23:13:19.067307 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:19.061964    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr" event={"ID":"e1c4d22b-f1d2-4609-992a-fca963ed46c5","Type":"ContainerDied","Data":"08eaaeb29ff10cac66287ebf3eb30815ff2a3caa066bfe54bdebc5a73c6a5464"}
Oct 30 23:13:19.067307 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:19.062166    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr" event={"ID":"e1c4d22b-f1d2-4609-992a-fca963ed46c5","Type":"ContainerDied","Data":"7c6a821d0f4d098ac139e0b86c799e70e117aa91bf5b2fd645f73cee57299b9b"}
Oct 30 23:13:19.067307 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:19.062201    4889 scope.go:122] "RemoveContainer" containerID="08eaaeb29ff10cac66287ebf3eb30815ff2a3caa066bfe54bdebc5a73c6a5464"
Oct 30 23:13:19.077884 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:19.077861    4889 scope.go:122] "RemoveContainer" containerID="08eaaeb29ff10cac66287ebf3eb30815ff2a3caa066bfe54bdebc5a73c6a5464"
Oct 30 23:13:19.078356 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:13:19.078317    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08eaaeb29ff10cac66287ebf3eb30815ff2a3caa066bfe54bdebc5a73c6a5464\": container with ID starting with 08eaaeb29ff10cac66287ebf3eb30815ff2a3caa066bfe54bdebc5a73c6a5464 not found: ID does not exist" containerID="08eaaeb29ff10cac66287ebf3eb30815ff2a3caa066bfe54bdebc5a73c6a5464"
Oct 30 23:13:19.078529 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:19.078364    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08eaaeb29ff10cac66287ebf3eb30815ff2a3caa066bfe54bdebc5a73c6a5464"} err="failed to get container status \"08eaaeb29ff10cac66287ebf3eb30815ff2a3caa066bfe54bdebc5a73c6a5464\": rpc error: code = NotFound desc = could not find container \"08eaaeb29ff10cac66287ebf3eb30815ff2a3caa066bfe54bdebc5a73c6a5464\": container with ID starting with 08eaaeb29ff10cac66287ebf3eb30815ff2a3caa066bfe54bdebc5a73c6a5464 not found: ID does not exist"
Oct 30 23:13:19.098525 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:19.098493    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"]
Oct 30 23:13:19.098748 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:19.098714    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj" podUID="833e155d-05c6-4615-acad-bda7aa964a5e" containerName="istio-proxy" containerID="cri-o://bcf09d81425c21393724b2f06c05529a96d0b4e73191af1ab990e22be3b6fef7" gracePeriod=30
Oct 30 23:13:19.136140 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:19.136112    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"]
Oct 30 23:13:19.142836 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:19.142806    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-2cad7e/simulator-single-main-2cad7e-kserve-6665bd8cb-k95hr"]
Oct 30 23:13:19.182941 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:19.182905    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:13:19.355355 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:19.355269    4889 patch_prober.go:29] interesting pod/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:13:19.355355 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:19.355336    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj" podUID="833e155d-05c6-4615-acad-bda7aa964a5e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:13:20.773536 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:20.773501    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1c4d22b-f1d2-4609-992a-fca963ed46c5" path="/var/lib/kubelet/pods/e1c4d22b-f1d2-4609-992a-fca963ed46c5/volumes"
Oct 30 23:13:21.627736 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:21.627697    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws" podUID="749bf3ff-6eba-4741-b072-ed34fdc62cf6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:13:22.632938 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:22.632898    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4" podUID="f1617c11-0158-43f0-ad92-c658f3264433" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:13:24.359102 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.359065    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:13:24.386193 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.386104    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-credential-socket\") pod \"833e155d-05c6-4615-acad-bda7aa964a5e\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") "
Oct 30 23:13:24.386193 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.386160    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/833e155d-05c6-4615-acad-bda7aa964a5e-kube-api-access-msstj\" (UniqueName: \"kubernetes.io/projected/833e155d-05c6-4615-acad-bda7aa964a5e-kube-api-access-msstj\") pod \"833e155d-05c6-4615-acad-bda7aa964a5e\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") "
Oct 30 23:13:24.386471 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.386196    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/833e155d-05c6-4615-acad-bda7aa964a5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/833e155d-05c6-4615-acad-bda7aa964a5e-istio-podinfo\") pod \"833e155d-05c6-4615-acad-bda7aa964a5e\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") "
Oct 30 23:13:24.386471 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.386271    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/833e155d-05c6-4615-acad-bda7aa964a5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/833e155d-05c6-4615-acad-bda7aa964a5e-istiod-ca-cert\") pod \"833e155d-05c6-4615-acad-bda7aa964a5e\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") "
Oct 30 23:13:24.386471 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.386304    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-workload-socket\") pod \"833e155d-05c6-4615-acad-bda7aa964a5e\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") "
Oct 30 23:13:24.386471 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.386359    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-istio-envoy\") pod \"833e155d-05c6-4615-acad-bda7aa964a5e\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") "
Oct 30 23:13:24.386471 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.386427    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-workload-certs\") pod \"833e155d-05c6-4615-acad-bda7aa964a5e\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") "
Oct 30 23:13:24.386471 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.386442    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-credential-socket" pod "833e155d-05c6-4615-acad-bda7aa964a5e" (UID: "833e155d-05c6-4615-acad-bda7aa964a5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:13:24.386790 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.386482    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/833e155d-05c6-4615-acad-bda7aa964a5e-istio-token\" (UniqueName: \"kubernetes.io/projected/833e155d-05c6-4615-acad-bda7aa964a5e-istio-token\") pod \"833e155d-05c6-4615-acad-bda7aa964a5e\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") "
Oct 30 23:13:24.386790 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.386529    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-istio-data\") pod \"833e155d-05c6-4615-acad-bda7aa964a5e\" (UID: \"833e155d-05c6-4615-acad-bda7aa964a5e\") "
Oct 30 23:13:24.386790 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.386636    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-workload-socket" pod "833e155d-05c6-4615-acad-bda7aa964a5e" (UID: "833e155d-05c6-4615-acad-bda7aa964a5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:13:24.386790 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.386735    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/833e155d-05c6-4615-acad-bda7aa964a5e-istiod-ca-cert" pod "833e155d-05c6-4615-acad-bda7aa964a5e" (UID: "833e155d-05c6-4615-acad-bda7aa964a5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:13:24.386957 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.386912    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-istio-data" pod "833e155d-05c6-4615-acad-bda7aa964a5e" (UID: "833e155d-05c6-4615-acad-bda7aa964a5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:13:24.387208 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.387184    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-workload-certs" pod "833e155d-05c6-4615-acad-bda7aa964a5e" (UID: "833e155d-05c6-4615-acad-bda7aa964a5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:13:24.387292 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.387272    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/833e155d-05c6-4615-acad-bda7aa964a5e-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:13:24.387351 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.387300    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:13:24.387351 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.387315    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:13:24.387351 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.387331    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:13:24.388796 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.388771    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/833e155d-05c6-4615-acad-bda7aa964a5e-istio-podinfo" pod "833e155d-05c6-4615-acad-bda7aa964a5e" (UID: "833e155d-05c6-4615-acad-bda7aa964a5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:13:24.388943 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.388798    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-istio-envoy" pod "833e155d-05c6-4615-acad-bda7aa964a5e" (UID: "833e155d-05c6-4615-acad-bda7aa964a5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:13:24.389290 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.389270    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833e155d-05c6-4615-acad-bda7aa964a5e-istio-token" pod "833e155d-05c6-4615-acad-bda7aa964a5e" (UID: "833e155d-05c6-4615-acad-bda7aa964a5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:13:24.389388 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.389295    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833e155d-05c6-4615-acad-bda7aa964a5e-kube-api-access-msstj" pod "833e155d-05c6-4615-acad-bda7aa964a5e" (UID: "833e155d-05c6-4615-acad-bda7aa964a5e"). InnerVolumeSpecName "kube-api-access-msstj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:13:24.487885 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.487848    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:13:24.487885 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.487881    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/833e155d-05c6-4615-acad-bda7aa964a5e-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:13:24.487885 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.487894    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/833e155d-05c6-4615-acad-bda7aa964a5e-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:13:24.488164 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.487905    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msstj\" (UniqueName: \"kubernetes.io/projected/833e155d-05c6-4615-acad-bda7aa964a5e-kube-api-access-msstj\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:13:24.488164 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:24.487917    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/833e155d-05c6-4615-acad-bda7aa964a5e-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:13:25.095054 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:25.094960    4889 generic.go:356] "Generic (PLEG): container finished" podID="833e155d-05c6-4615-acad-bda7aa964a5e" containerID="bcf09d81425c21393724b2f06c05529a96d0b4e73191af1ab990e22be3b6fef7" exitCode=0
Oct 30 23:13:25.095237 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:25.095080    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj" event={"ID":"833e155d-05c6-4615-acad-bda7aa964a5e","Type":"ContainerDied","Data":"bcf09d81425c21393724b2f06c05529a96d0b4e73191af1ab990e22be3b6fef7"}
Oct 30 23:13:25.095237 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:25.095099    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"
Oct 30 23:13:25.095237 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:25.095137    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj" event={"ID":"833e155d-05c6-4615-acad-bda7aa964a5e","Type":"ContainerDied","Data":"71cfd31cd40904d208f4b774b41e5364cefb38a171b1f195cd32e5a575d547bc"}
Oct 30 23:13:25.095237 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:25.095168    4889 scope.go:122] "RemoveContainer" containerID="bcf09d81425c21393724b2f06c05529a96d0b4e73191af1ab990e22be3b6fef7"
Oct 30 23:13:25.104607 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:25.104585    4889 scope.go:122] "RemoveContainer" containerID="6f8f79bea637ae4a91f136d29783a3423e59697f00bbb2dd316d930b76af83b9"
Oct 30 23:13:25.113667 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:25.113644    4889 scope.go:122] "RemoveContainer" containerID="bcf09d81425c21393724b2f06c05529a96d0b4e73191af1ab990e22be3b6fef7"
Oct 30 23:13:25.113973 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:13:25.113951    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcf09d81425c21393724b2f06c05529a96d0b4e73191af1ab990e22be3b6fef7\": container with ID starting with bcf09d81425c21393724b2f06c05529a96d0b4e73191af1ab990e22be3b6fef7 not found: ID does not exist" containerID="bcf09d81425c21393724b2f06c05529a96d0b4e73191af1ab990e22be3b6fef7"
Oct 30 23:13:25.114058 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:25.113989    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcf09d81425c21393724b2f06c05529a96d0b4e73191af1ab990e22be3b6fef7"} err="failed to get container status \"bcf09d81425c21393724b2f06c05529a96d0b4e73191af1ab990e22be3b6fef7\": rpc error: code = NotFound desc = could not find container \"bcf09d81425c21393724b2f06c05529a96d0b4e73191af1ab990e22be3b6fef7\": container with ID starting with bcf09d81425c21393724b2f06c05529a96d0b4e73191af1ab990e22be3b6fef7 not found: ID does not exist"
Oct 30 23:13:25.114058 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:25.114017    4889 scope.go:122] "RemoveContainer" containerID="6f8f79bea637ae4a91f136d29783a3423e59697f00bbb2dd316d930b76af83b9"
Oct 30 23:13:25.114324 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:13:25.114305    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f8f79bea637ae4a91f136d29783a3423e59697f00bbb2dd316d930b76af83b9\": container with ID starting with 6f8f79bea637ae4a91f136d29783a3423e59697f00bbb2dd316d930b76af83b9 not found: ID does not exist" containerID="6f8f79bea637ae4a91f136d29783a3423e59697f00bbb2dd316d930b76af83b9"
Oct 30 23:13:25.114368 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:25.114333    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f8f79bea637ae4a91f136d29783a3423e59697f00bbb2dd316d930b76af83b9"} err="failed to get container status \"6f8f79bea637ae4a91f136d29783a3423e59697f00bbb2dd316d930b76af83b9\": rpc error: code = NotFound desc = could not find container \"6f8f79bea637ae4a91f136d29783a3423e59697f00bbb2dd316d930b76af83b9\": container with ID starting with 6f8f79bea637ae4a91f136d29783a3423e59697f00bbb2dd316d930b76af83b9 not found: ID does not exist"
Oct 30 23:13:25.167568 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:25.167526    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"]
Oct 30 23:13:25.181038 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:25.180998    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2cad7e-openshift-default-6465775477q6cwj"]
Oct 30 23:13:26.773071 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:26.773040    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="833e155d-05c6-4615-acad-bda7aa964a5e" path="/var/lib/kubelet/pods/833e155d-05c6-4615-acad-bda7aa964a5e/volumes"
Oct 30 23:13:31.627720 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:31.627675    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws" podUID="749bf3ff-6eba-4741-b072-ed34fdc62cf6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:13:32.632832 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:32.632785    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4" podUID="f1617c11-0158-43f0-ad92-c658f3264433" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:13:34.148098 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:34.147927    4889 generic.go:356] "Generic (PLEG): container finished" podID="f1617c11-0158-43f0-ad92-c658f3264433" containerID="37fd75133cddf1482060e960d00c3f1686f7583ef8e833ce5af133de9d5359ba" exitCode=0
Oct 30 23:13:34.148098 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:34.148032    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4" event={"ID":"f1617c11-0158-43f0-ad92-c658f3264433","Type":"ContainerDied","Data":"37fd75133cddf1482060e960d00c3f1686f7583ef8e833ce5af133de9d5359ba"}
Oct 30 23:13:34.151192 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:34.151159    4889 generic.go:356] "Generic (PLEG): container finished" podID="749bf3ff-6eba-4741-b072-ed34fdc62cf6" containerID="2694add0ff6119dc384b7669e5254b3d63daae8e56395bb9de4c136cedb0f7e5" exitCode=0
Oct 30 23:13:34.151328 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:34.151304    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws" event={"ID":"749bf3ff-6eba-4741-b072-ed34fdc62cf6","Type":"ContainerDied","Data":"2694add0ff6119dc384b7669e5254b3d63daae8e56395bb9de4c136cedb0f7e5"}
Oct 30 23:13:34.231059 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:34.231029    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws"
Oct 30 23:13:34.255647 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:34.255627    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4"
Oct 30 23:13:34.290441 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:34.290414    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/749bf3ff-6eba-4741-b072-ed34fdc62cf6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749bf3ff-6eba-4741-b072-ed34fdc62cf6-plugins-config-volume\") pod \"749bf3ff-6eba-4741-b072-ed34fdc62cf6\" (UID: \"749bf3ff-6eba-4741-b072-ed34fdc62cf6\") "
Oct 30 23:13:34.290568 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:34.290458    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/749bf3ff-6eba-4741-b072-ed34fdc62cf6-kube-api-access-52ng9\" (UniqueName: \"kubernetes.io/projected/749bf3ff-6eba-4741-b072-ed34fdc62cf6-kube-api-access-52ng9\") pod \"749bf3ff-6eba-4741-b072-ed34fdc62cf6\" (UID: \"749bf3ff-6eba-4741-b072-ed34fdc62cf6\") "
Oct 30 23:13:34.290755 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:34.290735    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/749bf3ff-6eba-4741-b072-ed34fdc62cf6-plugins-config-volume" pod "749bf3ff-6eba-4741-b072-ed34fdc62cf6" (UID: "749bf3ff-6eba-4741-b072-ed34fdc62cf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:13:34.290853 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:34.290837    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749bf3ff-6eba-4741-b072-ed34fdc62cf6-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:13:34.292706 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:34.292683    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/749bf3ff-6eba-4741-b072-ed34fdc62cf6-kube-api-access-52ng9" pod "749bf3ff-6eba-4741-b072-ed34fdc62cf6" (UID: "749bf3ff-6eba-4741-b072-ed34fdc62cf6"). InnerVolumeSpecName "kube-api-access-52ng9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:13:34.391869 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:34.391839    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1617c11-0158-43f0-ad92-c658f3264433-kube-api-access-d64rg\" (UniqueName: \"kubernetes.io/projected/f1617c11-0158-43f0-ad92-c658f3264433-kube-api-access-d64rg\") pod \"f1617c11-0158-43f0-ad92-c658f3264433\" (UID: \"f1617c11-0158-43f0-ad92-c658f3264433\") "
Oct 30 23:13:34.392047 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:34.391944    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1617c11-0158-43f0-ad92-c658f3264433-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1617c11-0158-43f0-ad92-c658f3264433-plugins-config-volume\") pod \"f1617c11-0158-43f0-ad92-c658f3264433\" (UID: \"f1617c11-0158-43f0-ad92-c658f3264433\") "
Oct 30 23:13:34.392272 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:34.392213    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52ng9\" (UniqueName: \"kubernetes.io/projected/749bf3ff-6eba-4741-b072-ed34fdc62cf6-kube-api-access-52ng9\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:13:34.392404 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:34.392358    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1617c11-0158-43f0-ad92-c658f3264433-plugins-config-volume" pod "f1617c11-0158-43f0-ad92-c658f3264433" (UID: "f1617c11-0158-43f0-ad92-c658f3264433"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:13:34.394066 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:34.394042    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1617c11-0158-43f0-ad92-c658f3264433-kube-api-access-d64rg" pod "f1617c11-0158-43f0-ad92-c658f3264433" (UID: "f1617c11-0158-43f0-ad92-c658f3264433"). InnerVolumeSpecName "kube-api-access-d64rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:13:34.493544 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:34.493513    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1617c11-0158-43f0-ad92-c658f3264433-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:13:34.493544 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:34.493539    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d64rg\" (UniqueName: \"kubernetes.io/projected/f1617c11-0158-43f0-ad92-c658f3264433-kube-api-access-d64rg\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:13:35.156918 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:35.156834    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4"
Oct 30 23:13:35.157420 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:35.156844    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4" event={"ID":"f1617c11-0158-43f0-ad92-c658f3264433","Type":"ContainerDied","Data":"a565bdd72eba7fc1758ec8826c94273388aaa5a1024303ff2f2e66dd5d3e3e16"}
Oct 30 23:13:35.157420 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:35.156975    4889 scope.go:122] "RemoveContainer" containerID="37fd75133cddf1482060e960d00c3f1686f7583ef8e833ce5af133de9d5359ba"
Oct 30 23:13:35.158393 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:35.158322    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws"
Oct 30 23:13:35.158393 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:35.158366    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws" event={"ID":"749bf3ff-6eba-4741-b072-ed34fdc62cf6","Type":"ContainerDied","Data":"e39a3a23c479dd9332a6e3ef3b5846bc335e633b442fd1404d6c981b2ddeccf1"}
Oct 30 23:13:35.174859 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:35.174838    4889 scope.go:122] "RemoveContainer" containerID="2694add0ff6119dc384b7669e5254b3d63daae8e56395bb9de4c136cedb0f7e5"
Oct 30 23:13:35.213619 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:35.213593    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws"]
Oct 30 23:13:35.227085 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:35.227061    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2cad7e-758854df5pq2ws"]
Oct 30 23:13:35.255551 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:35.255522    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4"]
Oct 30 23:13:35.259714 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:35.259690    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cad7e-d85d5hddg4"]
Oct 30 23:13:36.773017 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:36.772983    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="749bf3ff-6eba-4741-b072-ed34fdc62cf6" path="/var/lib/kubelet/pods/749bf3ff-6eba-4741-b072-ed34fdc62cf6/volumes"
Oct 30 23:13:36.773411 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:36.773377    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1617c11-0158-43f0-ad92-c658f3264433" path="/var/lib/kubelet/pods/f1617c11-0158-43f0-ad92-c658f3264433/volumes"
Oct 30 23:13:39.958729 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:39.958694    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:13:40.301238 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:13:40.301182    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:13:41.249301 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:41.249259    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:13:47.113447 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:47.113407    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:13:52.896639 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:13:52.896589    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:14:10.300842 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:14:10.300805    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:14:40.301842 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:14:40.301802    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:14:53.032154 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:14:53.032120    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:14:58.956699 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:14:58.956660    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:15:10.300751 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:15:10.300660    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:15:38.067964 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.067927    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:15:38.391330 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.391211    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"]
Oct 30 23:15:38.391739 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.391711    4889 cpu_manager.go:414] "containerMap: removing container" podUID="833e155d-05c6-4615-acad-bda7aa964a5e" containerName="istio-proxy"
Oct 30 23:15:38.391921 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.391728    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="833e155d-05c6-4615-acad-bda7aa964a5e" containerName="istio-proxy"
Oct 30 23:15:38.391921 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.391798    4889 cpu_manager.go:414] "containerMap: removing container" podUID="92783ce8-6296-418c-90d3-1807a1a8a3a0" containerName="maas-api"
Oct 30 23:15:38.391921 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.391808    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92783ce8-6296-418c-90d3-1807a1a8a3a0" containerName="maas-api"
Oct 30 23:15:38.391921 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.391830    4889 cpu_manager.go:414] "containerMap: removing container" podUID="749bf3ff-6eba-4741-b072-ed34fdc62cf6" containerName="payload-processing"
Oct 30 23:15:38.391921 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.391838    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="749bf3ff-6eba-4741-b072-ed34fdc62cf6" containerName="payload-processing"
Oct 30 23:15:38.391921 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.391848    4889 cpu_manager.go:414] "containerMap: removing container" podUID="833e155d-05c6-4615-acad-bda7aa964a5e" containerName="istio-proxy"
Oct 30 23:15:38.391921 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.391855    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="833e155d-05c6-4615-acad-bda7aa964a5e" containerName="istio-proxy"
Oct 30 23:15:38.391921 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.391874    4889 cpu_manager.go:414] "containerMap: removing container" podUID="f1617c11-0158-43f0-ad92-c658f3264433" containerName="payload-pre-processing"
Oct 30 23:15:38.391921 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.391882    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1617c11-0158-43f0-ad92-c658f3264433" containerName="payload-pre-processing"
Oct 30 23:15:38.391921 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.391894    4889 cpu_manager.go:414] "containerMap: removing container" podUID="e1c4d22b-f1d2-4609-992a-fca963ed46c5" containerName="main"
Oct 30 23:15:38.391921 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.391899    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1c4d22b-f1d2-4609-992a-fca963ed46c5" containerName="main"
Oct 30 23:15:38.392627 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.391987    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="749bf3ff-6eba-4741-b072-ed34fdc62cf6" containerName="payload-processing"
Oct 30 23:15:38.392627 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.391996    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="833e155d-05c6-4615-acad-bda7aa964a5e" containerName="istio-proxy"
Oct 30 23:15:38.392627 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.392008    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1617c11-0158-43f0-ad92-c658f3264433" containerName="payload-pre-processing"
Oct 30 23:15:38.392627 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.392016    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1c4d22b-f1d2-4609-992a-fca963ed46c5" containerName="main"
Oct 30 23:15:38.392627 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.392033    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="92783ce8-6296-418c-90d3-1807a1a8a3a0" containerName="maas-api"
Oct 30 23:15:38.395485 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.395465    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.414868 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.414832    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-ef9544-openshift-default-dockercfg-6vvfz\""
Oct 30 23:15:38.430276 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.430240    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"]
Oct 30 23:15:38.534638 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.534490    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-istio-envoy\") pod \"e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") " pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.534958 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.534925    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/258a920d-b0b1-4eae-9437-4289516d310a-istiod-ca-cert\") pod \"e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") " pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.535118 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.535006    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/258a920d-b0b1-4eae-9437-4289516d310a-istio-token\") pod \"e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") " pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.535118 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.535037    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-workload-certs\") pod \"e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") " pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.535264 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.535162    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-workload-socket\") pod \"e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") " pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.535264 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.535211    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rg96\" (UniqueName: \"kubernetes.io/projected/258a920d-b0b1-4eae-9437-4289516d310a-kube-api-access-9rg96\") pod \"e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") " pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.535384 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.535285    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/258a920d-b0b1-4eae-9437-4289516d310a-istio-podinfo\") pod \"e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") " pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.535384 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.535350    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-istio-data\") pod \"e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") " pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.535500 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.535391    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-credential-socket\") pod \"e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") " pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.536587 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.536552    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:15:38.636552 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.636511    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-istio-data\") pod \"e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") " pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.636794 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.636560    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-credential-socket\") pod \"e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") " pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.636794 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.636606    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-istio-envoy\") pod \"e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") " pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.636794 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.636642    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/258a920d-b0b1-4eae-9437-4289516d310a-istiod-ca-cert\") pod \"e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") " pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.636794 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.636700    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/258a920d-b0b1-4eae-9437-4289516d310a-istio-token\") pod \"e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") " pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.636794 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.636722    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-workload-certs\") pod \"e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") " pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.636794 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.636763    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-workload-socket\") pod \"e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") " pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.636794 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.636794    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rg96\" (UniqueName: \"kubernetes.io/projected/258a920d-b0b1-4eae-9437-4289516d310a-kube-api-access-9rg96\") pod \"e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") " pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.637302 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.636838    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/258a920d-b0b1-4eae-9437-4289516d310a-istio-podinfo\") pod \"e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") " pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.637302 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.637038    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-istio-data\") pod \"e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") " pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.637302 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.637121    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-workload-certs\") pod \"e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") " pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.637435 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.637309    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-credential-socket\") pod \"e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") " pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.637435 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.637395    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-workload-socket\") pod \"e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") " pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.637520 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.637479    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/258a920d-b0b1-4eae-9437-4289516d310a-istiod-ca-cert\") pod \"e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") " pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.639322 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.639301    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-istio-envoy\") pod \"e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") " pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.639454 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.639437    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/258a920d-b0b1-4eae-9437-4289516d310a-istio-podinfo\") pod \"e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") " pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.658399 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.658319    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/258a920d-b0b1-4eae-9437-4289516d310a-istio-token\") pod \"e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") " pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.658639 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.658618    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rg96\" (UniqueName: \"kubernetes.io/projected/258a920d-b0b1-4eae-9437-4289516d310a-kube-api-access-9rg96\") pod \"e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") " pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.709408 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.709367    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:38.715168 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:38.715143    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:15:39.025571 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:39.025536    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"]
Oct 30 23:15:39.026944 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:15:39.026899    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod258a920d_b0b1_4eae_9437_4289516d310a.slice/crio-9df6c248c8f8e69ac7268565f3a939c9c02070c821b3b8f937851c821c9d8d8b WatchSource:0}: Error finding container 9df6c248c8f8e69ac7268565f3a939c9c02070c821b3b8f937851c821c9d8d8b: Status 404 returned error can't find the container with id 9df6c248c8f8e69ac7268565f3a939c9c02070c821b3b8f937851c821c9d8d8b
Oct 30 23:15:39.029344 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:39.029311    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:15:39.029464 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:39.029394    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:15:39.029464 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:39.029439    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:15:39.806987 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:39.806942    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l" event={"ID":"258a920d-b0b1-4eae-9437-4289516d310a","Type":"ContainerStarted","Data":"54c4651de56eeb3868e6462aba6a08d4d99103616ae545c7eea2e7cb55645ead"}
Oct 30 23:15:39.807465 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:39.807004    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l" event={"ID":"258a920d-b0b1-4eae-9437-4289516d310a","Type":"ContainerStarted","Data":"9df6c248c8f8e69ac7268565f3a939c9c02070c821b3b8f937851c821c9d8d8b"}
Oct 30 23:15:39.866819 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:39.866760    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l" podStartSLOduration=1.86674389 podStartE2EDuration="1.86674389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:15:39.86513675 +0000 UTC m=+4125.221503065" watchObservedRunningTime="2026-10-30 23:15:39.86674389 +0000 UTC m=+4125.223110239"
Oct 30 23:15:40.301978 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:15:40.301936    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:15:40.710438 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:40.710362    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:40.715640 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:40.715613    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:40.813023 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:40.812989    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:40.814017 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:40.813997    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:42.316778 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:42.316735    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:15:42.493388 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:42.493352    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:15:43.698966 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:43.698931    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw"]
Oct 30 23:15:43.699454 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:43.699439    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="833e155d-05c6-4615-acad-bda7aa964a5e" containerName="istio-proxy"
Oct 30 23:15:43.702744 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:43.702712    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw"
Oct 30 23:15:43.733406 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:43.733376    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-ef9544-serving-cert\""
Oct 30 23:15:43.735919 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:43.735893    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw"]
Oct 30 23:15:43.789933 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:43.789892    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dplt\" (UniqueName: \"kubernetes.io/projected/2d9edca1-e32c-407e-ac4b-22e1736966ac-kube-api-access-6dplt\") pod \"maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw\" (UID: \"2d9edca1-e32c-407e-ac4b-22e1736966ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw"
Oct 30 23:15:43.789933 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:43.789939    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9edca1-e32c-407e-ac4b-22e1736966ac-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw\" (UID: \"2d9edca1-e32c-407e-ac4b-22e1736966ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw"
Oct 30 23:15:43.790204 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:43.790083    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d9edca1-e32c-407e-ac4b-22e1736966ac-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw\" (UID: \"2d9edca1-e32c-407e-ac4b-22e1736966ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw"
Oct 30 23:15:43.790204 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:43.790126    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d9edca1-e32c-407e-ac4b-22e1736966ac-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw\" (UID: \"2d9edca1-e32c-407e-ac4b-22e1736966ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw"
Oct 30 23:15:43.844253 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:43.844205    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:15:43.891335 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:43.891296    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d9edca1-e32c-407e-ac4b-22e1736966ac-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw\" (UID: \"2d9edca1-e32c-407e-ac4b-22e1736966ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw"
Oct 30 23:15:43.891680 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:43.891648    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d9edca1-e32c-407e-ac4b-22e1736966ac-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw\" (UID: \"2d9edca1-e32c-407e-ac4b-22e1736966ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw"
Oct 30 23:15:43.892032 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:43.891818    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dplt\" (UniqueName: \"kubernetes.io/projected/2d9edca1-e32c-407e-ac4b-22e1736966ac-kube-api-access-6dplt\") pod \"maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw\" (UID: \"2d9edca1-e32c-407e-ac4b-22e1736966ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw"
Oct 30 23:15:43.892032 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:43.891867    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9edca1-e32c-407e-ac4b-22e1736966ac-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw\" (UID: \"2d9edca1-e32c-407e-ac4b-22e1736966ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw"
Oct 30 23:15:43.892741 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:43.892716    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9edca1-e32c-407e-ac4b-22e1736966ac-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw\" (UID: \"2d9edca1-e32c-407e-ac4b-22e1736966ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw"
Oct 30 23:15:43.894947 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:43.894921    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d9edca1-e32c-407e-ac4b-22e1736966ac-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw\" (UID: \"2d9edca1-e32c-407e-ac4b-22e1736966ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw"
Oct 30 23:15:43.895420 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:43.895400    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d9edca1-e32c-407e-ac4b-22e1736966ac-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw\" (UID: \"2d9edca1-e32c-407e-ac4b-22e1736966ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw"
Oct 30 23:15:43.917847 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:43.917815    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dplt\" (UniqueName: \"kubernetes.io/projected/2d9edca1-e32c-407e-ac4b-22e1736966ac-kube-api-access-6dplt\") pod \"maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw\" (UID: \"2d9edca1-e32c-407e-ac4b-22e1736966ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw"
Oct 30 23:15:44.018356 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.018317    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw"
Oct 30 23:15:44.022299 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.022259    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:15:44.181336 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.181302    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67"]
Oct 30 23:15:44.186041 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.186024    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67"
Oct 30 23:15:44.194399 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.194374    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4mqf\" (UniqueName: \"kubernetes.io/projected/0650ad04-1155-4626-8480-53e7e3d82e39-kube-api-access-h4mqf\") pod \"payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67\" (UID: \"0650ad04-1155-4626-8480-53e7e3d82e39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67"
Oct 30 23:15:44.194533 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.194427    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0650ad04-1155-4626-8480-53e7e3d82e39-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67\" (UID: \"0650ad04-1155-4626-8480-53e7e3d82e39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67"
Oct 30 23:15:44.203865 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.203835    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67"]
Oct 30 23:15:44.204027 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.203976    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-ef9544\""
Oct 30 23:15:44.204104 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.204031    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-40496db4-dockercfg-8vsg4\""
Oct 30 23:15:44.205350 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.205322    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:15:44.289216 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.284002    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9"]
Oct 30 23:15:44.289216 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.288415    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9"
Oct 30 23:15:44.295832 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.295800    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0650ad04-1155-4626-8480-53e7e3d82e39-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67\" (UID: \"0650ad04-1155-4626-8480-53e7e3d82e39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67"
Oct 30 23:15:44.295998 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.295896    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67976ad6-64a0-43aa-8167-dad668fdcb4e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9\" (UID: \"67976ad6-64a0-43aa-8167-dad668fdcb4e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9"
Oct 30 23:15:44.295998 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.295972    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4mqf\" (UniqueName: \"kubernetes.io/projected/0650ad04-1155-4626-8480-53e7e3d82e39-kube-api-access-h4mqf\") pod \"payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67\" (UID: \"0650ad04-1155-4626-8480-53e7e3d82e39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67"
Oct 30 23:15:44.295998 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.295995    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jts5x\" (UniqueName: \"kubernetes.io/projected/67976ad6-64a0-43aa-8167-dad668fdcb4e-kube-api-access-jts5x\") pod \"payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9\" (UID: \"67976ad6-64a0-43aa-8167-dad668fdcb4e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9"
Oct 30 23:15:44.297324 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.296966    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0650ad04-1155-4626-8480-53e7e3d82e39-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67\" (UID: \"0650ad04-1155-4626-8480-53e7e3d82e39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67"
Oct 30 23:15:44.299840 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.299812    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9"]
Oct 30 23:15:44.317843 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.317805    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4mqf\" (UniqueName: \"kubernetes.io/projected/0650ad04-1155-4626-8480-53e7e3d82e39-kube-api-access-h4mqf\") pod \"payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67\" (UID: \"0650ad04-1155-4626-8480-53e7e3d82e39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67"
Oct 30 23:15:44.348487 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.348458    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw"]
Oct 30 23:15:44.349384 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:15:44.349360    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9edca1_e32c_407e_ac4b_22e1736966ac.slice/crio-e2b9bdc6b130d21674066295a9ff3236b27a9e7254de4c2861936b5c12263631 WatchSource:0}: Error finding container e2b9bdc6b130d21674066295a9ff3236b27a9e7254de4c2861936b5c12263631: Status 404 returned error can't find the container with id e2b9bdc6b130d21674066295a9ff3236b27a9e7254de4c2861936b5c12263631
Oct 30 23:15:44.397462 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.397421    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jts5x\" (UniqueName: \"kubernetes.io/projected/67976ad6-64a0-43aa-8167-dad668fdcb4e-kube-api-access-jts5x\") pod \"payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9\" (UID: \"67976ad6-64a0-43aa-8167-dad668fdcb4e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9"
Oct 30 23:15:44.397669 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.397533    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67976ad6-64a0-43aa-8167-dad668fdcb4e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9\" (UID: \"67976ad6-64a0-43aa-8167-dad668fdcb4e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9"
Oct 30 23:15:44.398279 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.398261    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67976ad6-64a0-43aa-8167-dad668fdcb4e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9\" (UID: \"67976ad6-64a0-43aa-8167-dad668fdcb4e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9"
Oct 30 23:15:44.418911 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.418883    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jts5x\" (UniqueName: \"kubernetes.io/projected/67976ad6-64a0-43aa-8167-dad668fdcb4e-kube-api-access-jts5x\") pod \"payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9\" (UID: \"67976ad6-64a0-43aa-8167-dad668fdcb4e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9"
Oct 30 23:15:44.499729 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.499690    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67"
Oct 30 23:15:44.605768 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.605726    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9"
Oct 30 23:15:44.808692 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.808663    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67"]
Oct 30 23:15:44.809679 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:15:44.809655    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0650ad04_1155_4626_8480_53e7e3d82e39.slice/crio-2f83b7882d8e1724f53b8296423211d74600a09b80694264b1c822b11ec453c5 WatchSource:0}: Error finding container 2f83b7882d8e1724f53b8296423211d74600a09b80694264b1c822b11ec453c5: Status 404 returned error can't find the container with id 2f83b7882d8e1724f53b8296423211d74600a09b80694264b1c822b11ec453c5
Oct 30 23:15:44.840267 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.840195    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw" event={"ID":"2d9edca1-e32c-407e-ac4b-22e1736966ac","Type":"ContainerStarted","Data":"d924930151d9565670cb7ce088cf3ee8dc3ba73f55d92484fc7f7c34a0a9c107"}
Oct 30 23:15:44.840530 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.840290    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw" event={"ID":"2d9edca1-e32c-407e-ac4b-22e1736966ac","Type":"ContainerStarted","Data":"e2b9bdc6b130d21674066295a9ff3236b27a9e7254de4c2861936b5c12263631"}
Oct 30 23:15:44.843420 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.842921    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67" event={"ID":"0650ad04-1155-4626-8480-53e7e3d82e39","Type":"ContainerStarted","Data":"2f83b7882d8e1724f53b8296423211d74600a09b80694264b1c822b11ec453c5"}
Oct 30 23:15:44.889795 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.889748    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw" podStartSLOduration=1.565128767 podStartE2EDuration="1.889735669s" totalImagesPullingTime="324.606902ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:15:44.889366595 +0000 UTC m=+4130.245732925" watchObservedRunningTime="2026-10-30 23:15:44.889735669 +0000 UTC m=+4130.246101981"
Oct 30 23:15:44.906276 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:44.906245    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9"]
Oct 30 23:15:44.907404 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:15:44.907379    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67976ad6_64a0_43aa_8167_dad668fdcb4e.slice/crio-a9260343ccd3bc10bf1aa1ba674e8ebc9d0b28fb36e6cc4b5faeace8ed23559f WatchSource:0}: Error finding container a9260343ccd3bc10bf1aa1ba674e8ebc9d0b28fb36e6cc4b5faeace8ed23559f: Status 404 returned error can't find the container with id a9260343ccd3bc10bf1aa1ba674e8ebc9d0b28fb36e6cc4b5faeace8ed23559f
Oct 30 23:15:45.631749 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:45.631658    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:15:45.849974 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:45.849935    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67" event={"ID":"0650ad04-1155-4626-8480-53e7e3d82e39","Type":"ContainerStarted","Data":"4496e9250db477a7d979cf80b791329794f5bd859817037e87b24088ac4c8082"}
Oct 30 23:15:45.850410 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:45.850068    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67"
Oct 30 23:15:45.851643 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:45.851618    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9" event={"ID":"67976ad6-64a0-43aa-8167-dad668fdcb4e","Type":"ContainerStarted","Data":"4bfbac3263973641e17af7541624946e70557e72ba2a9c7b4eaa4198f383bb0b"}
Oct 30 23:15:45.851792 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:45.851661    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9" event={"ID":"67976ad6-64a0-43aa-8167-dad668fdcb4e","Type":"ContainerStarted","Data":"a9260343ccd3bc10bf1aa1ba674e8ebc9d0b28fb36e6cc4b5faeace8ed23559f"}
Oct 30 23:15:45.851792 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:45.851764    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw"
Oct 30 23:15:45.851920 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:45.851807    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9"
Oct 30 23:15:45.894322 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:45.894184    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67" podStartSLOduration=1.8941709389999999 podStartE2EDuration="1.894170939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:15:45.893408623 +0000 UTC m=+4131.249774938" watchObservedRunningTime="2026-10-30 23:15:45.894170939 +0000 UTC m=+4131.250537254"
Oct 30 23:15:45.936886 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:45.936832    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9" podStartSLOduration=1.936813782 podStartE2EDuration="1.936813782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:15:45.935165507 +0000 UTC m=+4131.291531821" watchObservedRunningTime="2026-10-30 23:15:45.936813782 +0000 UTC m=+4131.293180097"
Oct 30 23:15:47.400200 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:47.400154    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:15:49.235351 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:49.235299    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:15:50.937248 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:50.936780    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:15:51.862568 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:51.862536    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw"
Oct 30 23:15:52.665779 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:52.665740    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:15:53.135370 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:53.135330    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:15:54.646664 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:54.646628    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:15:55.124439 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:55.124400    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:15:56.234636 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:56.234598    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:15:56.584586 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:56.583671    4889 patch_prober.go:29] interesting pod/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": read tcp 10.132.0.2:55174->10.132.1.18:15021: read: connection reset by peer" start-of-body=
Oct 30 23:15:56.584586 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:56.583749    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l" podUID="258a920d-b0b1-4eae-9437-4289516d310a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": read tcp 10.132.0.2:55174->10.132.1.18:15021: read: connection reset by peer"
Oct 30 23:15:56.860840 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:56.860751    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67"
Oct 30 23:15:56.861148 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:56.861130    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9"
Oct 30 23:15:56.909388 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:56.909356    4889 generic.go:356] "Generic (PLEG): container finished" podID="258a920d-b0b1-4eae-9437-4289516d310a" containerID="54c4651de56eeb3868e6462aba6a08d4d99103616ae545c7eea2e7cb55645ead" exitCode=0
Oct 30 23:15:56.909575 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:56.909423    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l" event={"ID":"258a920d-b0b1-4eae-9437-4289516d310a","Type":"ContainerDied","Data":"54c4651de56eeb3868e6462aba6a08d4d99103616ae545c7eea2e7cb55645ead"}
Oct 30 23:15:56.909810 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:56.909797    4889 scope.go:122] "RemoveContainer" containerID="54c4651de56eeb3868e6462aba6a08d4d99103616ae545c7eea2e7cb55645ead"
Oct 30 23:15:56.911009 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:56.910983    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:15:56.911070 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:56.911046    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:15:56.911106 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:56.911077    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:15:57.222800 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:57.222705    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:15:57.843262 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:57.843210    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:15:57.939280 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:57.939211    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l" event={"ID":"258a920d-b0b1-4eae-9437-4289516d310a","Type":"ContainerStarted","Data":"1cca7a060eab06d9ce5fa506b2daf12bca7a83f6079aed10b6bbcb72ff186f5e"}
Oct 30 23:15:58.029650 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:58.029564    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:15:58.731767 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:58.731725    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:15:58.734406 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:58.734364    4889 patch_prober.go:29] interesting pod/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body=
Oct 30 23:15:58.734586 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:58.734437    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l" podUID="258a920d-b0b1-4eae-9437-4289516d310a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused"
Oct 30 23:15:59.107949 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.107907    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"]
Oct 30 23:15:59.137596 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.137561    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"
Oct 30 23:15:59.143197 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.143160    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"]
Oct 30 23:15:59.153747 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.153707    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-model-cache\") pod \"simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z\" (UID: \"163e6247-764e-4fcb-af26-f8896c00da5a\") " pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"
Oct 30 23:15:59.153938 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.153799    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r48sm\" (UniqueName: \"kubernetes.io/projected/163e6247-764e-4fcb-af26-f8896c00da5a-kube-api-access-r48sm\") pod \"simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z\" (UID: \"163e6247-764e-4fcb-af26-f8896c00da5a\") " pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"
Oct 30 23:15:59.153938 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.153841    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-dshm\") pod \"simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z\" (UID: \"163e6247-764e-4fcb-af26-f8896c00da5a\") " pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"
Oct 30 23:15:59.153938 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.153886    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-tmp-dir\") pod \"simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z\" (UID: \"163e6247-764e-4fcb-af26-f8896c00da5a\") " pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"
Oct 30 23:15:59.154075 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.154032    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-home\") pod \"simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z\" (UID: \"163e6247-764e-4fcb-af26-f8896c00da5a\") " pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"
Oct 30 23:15:59.154075 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.154057    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/163e6247-764e-4fcb-af26-f8896c00da5a-tls-certs\") pod \"simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z\" (UID: \"163e6247-764e-4fcb-af26-f8896c00da5a\") " pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"
Oct 30 23:15:59.169011 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.168957    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ef9544\"/\"default-dockercfg-l985m\""
Oct 30 23:15:59.169011 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.168966    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ef9544\"/\"simulator-single-main-ef9544-kserve-self-signed-certs\""
Oct 30 23:15:59.169277 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.169029    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ef9544\"/\"kube-root-ca.crt\""
Oct 30 23:15:59.169377 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.169358    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ef9544\"/\"openshift-service-ca.crt\""
Oct 30 23:15:59.252076 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.252033    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:15:59.255273 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.255214    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-model-cache\") pod \"simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z\" (UID: \"163e6247-764e-4fcb-af26-f8896c00da5a\") " pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"
Oct 30 23:15:59.255430 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.255294    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r48sm\" (UniqueName: \"kubernetes.io/projected/163e6247-764e-4fcb-af26-f8896c00da5a-kube-api-access-r48sm\") pod \"simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z\" (UID: \"163e6247-764e-4fcb-af26-f8896c00da5a\") " pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"
Oct 30 23:15:59.255430 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.255335    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-dshm\") pod \"simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z\" (UID: \"163e6247-764e-4fcb-af26-f8896c00da5a\") " pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"
Oct 30 23:15:59.255430 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.255397    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-tmp-dir\") pod \"simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z\" (UID: \"163e6247-764e-4fcb-af26-f8896c00da5a\") " pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"
Oct 30 23:15:59.255605 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.255476    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-home\") pod \"simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z\" (UID: \"163e6247-764e-4fcb-af26-f8896c00da5a\") " pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"
Oct 30 23:15:59.255605 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.255502    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/163e6247-764e-4fcb-af26-f8896c00da5a-tls-certs\") pod \"simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z\" (UID: \"163e6247-764e-4fcb-af26-f8896c00da5a\") " pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"
Oct 30 23:15:59.255605 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.255592    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-model-cache\") pod \"simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z\" (UID: \"163e6247-764e-4fcb-af26-f8896c00da5a\") " pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"
Oct 30 23:15:59.255838 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.255807    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-tmp-dir\") pod \"simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z\" (UID: \"163e6247-764e-4fcb-af26-f8896c00da5a\") " pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"
Oct 30 23:15:59.255838 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.255820    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-home\") pod \"simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z\" (UID: \"163e6247-764e-4fcb-af26-f8896c00da5a\") " pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"
Oct 30 23:15:59.258028 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.258003    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-dshm\") pod \"simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z\" (UID: \"163e6247-764e-4fcb-af26-f8896c00da5a\") " pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"
Oct 30 23:15:59.258360 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.258337    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/163e6247-764e-4fcb-af26-f8896c00da5a-tls-certs\") pod \"simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z\" (UID: \"163e6247-764e-4fcb-af26-f8896c00da5a\") " pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"
Oct 30 23:15:59.277285 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.277248    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r48sm\" (UniqueName: \"kubernetes.io/projected/163e6247-764e-4fcb-af26-f8896c00da5a-kube-api-access-r48sm\") pod \"simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z\" (UID: \"163e6247-764e-4fcb-af26-f8896c00da5a\") " pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"
Oct 30 23:15:59.453878 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.453775    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"
Oct 30 23:15:59.734311 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.711718    4889 patch_prober.go:29] interesting pod/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body=
Oct 30 23:15:59.734311 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.711779    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l" podUID="258a920d-b0b1-4eae-9437-4289516d310a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused"
Oct 30 23:15:59.764012 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.763984    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"]
Oct 30 23:15:59.765157 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:15:59.765122    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod163e6247_764e_4fcb_af26_f8896c00da5a.slice/crio-074b25921200160802bdd8c4131a264388a888030d6c419fe98dff999294a691 WatchSource:0}: Error finding container 074b25921200160802bdd8c4131a264388a888030d6c419fe98dff999294a691: Status 404 returned error can't find the container with id 074b25921200160802bdd8c4131a264388a888030d6c419fe98dff999294a691
Oct 30 23:15:59.853525 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.853438    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:15:59.952099 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.952062    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z" event={"ID":"163e6247-764e-4fcb-af26-f8896c00da5a","Type":"ContainerStarted","Data":"009c8ffdb2452b01be97d38db4c08ce180581d5fcd88cf1ddd4c9b91830a376d"}
Oct 30 23:15:59.952308 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.952112    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z" event={"ID":"163e6247-764e-4fcb-af26-f8896c00da5a","Type":"ContainerStarted","Data":"074b25921200160802bdd8c4131a264388a888030d6c419fe98dff999294a691"}
Oct 30 23:15:59.996017 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:15:59.995959    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z" podStartSLOduration=0.995944102 podStartE2EDuration="995.944102ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:15:59.994406723 +0000 UTC m=+4145.350773043" watchObservedRunningTime="2026-10-30 23:15:59.995944102 +0000 UTC m=+4145.352310417"
Oct 30 23:16:00.454553 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:00.454461    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"
Oct 30 23:16:00.721309 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:00.715706    4889 patch_prober.go:29] interesting pod/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body=
Oct 30 23:16:00.721309 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:00.715768    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l" podUID="258a920d-b0b1-4eae-9437-4289516d310a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused"
Oct 30 23:16:01.648248 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:01.648076    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:01.735757 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:01.735713    4889 patch_prober.go:29] interesting pod/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body=
Oct 30 23:16:01.736102 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:01.736060    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l" podUID="258a920d-b0b1-4eae-9437-4289516d310a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused"
Oct 30 23:16:02.715730 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:02.715685    4889 patch_prober.go:29] interesting pod/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body=
Oct 30 23:16:02.716418 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:02.716380    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l" podUID="258a920d-b0b1-4eae-9437-4289516d310a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused"
Oct 30 23:16:03.348557 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:03.348517    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:03.714189 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:03.714104    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:16:03.714575 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:03.714550    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:16:03.715497 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:03.715478    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:16:05.018895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:05.018857    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:06.711385 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:06.711351    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:08.330844 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:08.330808    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:09.454279 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:09.454234    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"
Oct 30 23:16:09.467756 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:09.467726    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"
Oct 30 23:16:10.018421 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:10.018369    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:10.022387 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:10.022360    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"
Oct 30 23:16:10.302739 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:16:10.302647    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:16:11.655842 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:11.655805    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:13.243845 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:13.243810    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:14.911371 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:14.911334    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:16.622707 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:16.622672    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:18.302733 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:18.302692    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:19.998765 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:19.998726    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:21.623058 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:21.623029    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:23.217664 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:23.217628    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:24.913025 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:24.912984    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:26.632093 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:26.632062    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:28.251416 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:28.251381    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:29.820015 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:29.819957    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:31.561115 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:31.561071    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:33.127212 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:33.127166    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:34.910905 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:34.910867    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:36.724125 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:36.724082    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:38.229181 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:38.229138    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:39.904672 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:39.904633    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:40.301247 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:16:40.301185    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:16:41.510491 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:41.510459    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:42.743345 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:42.743308    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:43.270445 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:43.270407    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:45.228508 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:45.228469    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:46.947877 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:46.947829    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:47.705852 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:47.705819    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"]
Oct 30 23:16:47.836616 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:47.836577    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:48.715453 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:48.715417    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:50.408356 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:50.408321    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:52.111074 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:52.111021    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:53.233519 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:53.233486    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:53.390057 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:53.390027    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9"]
Oct 30 23:16:53.422406 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:53.422371    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67"]
Oct 30 23:16:53.587834 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:53.587798    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:53.755198 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:53.755149    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:54.922071 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:54.922031    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw"]
Oct 30 23:16:54.922582 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:54.922260    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw" podUID="2d9edca1-e32c-407e-ac4b-22e1736966ac" containerName="maas-api" containerID="cri-o://d924930151d9565670cb7ce088cf3ee8dc3ba73f55d92484fc7f7c34a0a9c107" gracePeriod=30
Oct 30 23:16:55.042095 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:55.042040    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:55.512142 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:55.512108    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:55.689962 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:55.689922    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:55.880762 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:55.880666    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 23:16:55.890407 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:55.890382    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 23:16:56.157976 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:56.157943    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:16:56.170776 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:56.170757    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw"
Oct 30 23:16:56.235686 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:56.235652    4889 generic.go:356] "Generic (PLEG): container finished" podID="2d9edca1-e32c-407e-ac4b-22e1736966ac" containerID="d924930151d9565670cb7ce088cf3ee8dc3ba73f55d92484fc7f7c34a0a9c107" exitCode=0
Oct 30 23:16:56.235886 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:56.235697    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw"
Oct 30 23:16:56.235886 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:56.235762    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw" event={"ID":"2d9edca1-e32c-407e-ac4b-22e1736966ac","Type":"ContainerDied","Data":"d924930151d9565670cb7ce088cf3ee8dc3ba73f55d92484fc7f7c34a0a9c107"}
Oct 30 23:16:56.235886 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:56.235836    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw" event={"ID":"2d9edca1-e32c-407e-ac4b-22e1736966ac","Type":"ContainerDied","Data":"e2b9bdc6b130d21674066295a9ff3236b27a9e7254de4c2861936b5c12263631"}
Oct 30 23:16:56.235886 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:56.235866    4889 scope.go:122] "RemoveContainer" containerID="d924930151d9565670cb7ce088cf3ee8dc3ba73f55d92484fc7f7c34a0a9c107"
Oct 30 23:16:56.237196 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:56.237178    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d9edca1-e32c-407e-ac4b-22e1736966ac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d9edca1-e32c-407e-ac4b-22e1736966ac-maas-api-tls\") pod \"2d9edca1-e32c-407e-ac4b-22e1736966ac\" (UID: \"2d9edca1-e32c-407e-ac4b-22e1736966ac\") "
Oct 30 23:16:56.237325 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:56.237307    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d9edca1-e32c-407e-ac4b-22e1736966ac-kube-api-access-6dplt\" (UniqueName: \"kubernetes.io/projected/2d9edca1-e32c-407e-ac4b-22e1736966ac-kube-api-access-6dplt\") pod \"2d9edca1-e32c-407e-ac4b-22e1736966ac\" (UID: \"2d9edca1-e32c-407e-ac4b-22e1736966ac\") "
Oct 30 23:16:56.237384 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:56.237370    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d9edca1-e32c-407e-ac4b-22e1736966ac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9edca1-e32c-407e-ac4b-22e1736966ac-openshift-service-ca\") pod \"2d9edca1-e32c-407e-ac4b-22e1736966ac\" (UID: \"2d9edca1-e32c-407e-ac4b-22e1736966ac\") "
Oct 30 23:16:56.237440 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:56.237423    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d9edca1-e32c-407e-ac4b-22e1736966ac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d9edca1-e32c-407e-ac4b-22e1736966ac-metrics-certs\") pod \"2d9edca1-e32c-407e-ac4b-22e1736966ac\" (UID: \"2d9edca1-e32c-407e-ac4b-22e1736966ac\") "
Oct 30 23:16:56.238205 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:56.238181    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9edca1-e32c-407e-ac4b-22e1736966ac-openshift-service-ca" pod "2d9edca1-e32c-407e-ac4b-22e1736966ac" (UID: "2d9edca1-e32c-407e-ac4b-22e1736966ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:16:56.240137 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:56.240111    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d9edca1-e32c-407e-ac4b-22e1736966ac-maas-api-tls" pod "2d9edca1-e32c-407e-ac4b-22e1736966ac" (UID: "2d9edca1-e32c-407e-ac4b-22e1736966ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:16:56.240817 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:56.240774    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d9edca1-e32c-407e-ac4b-22e1736966ac-metrics-certs" pod "2d9edca1-e32c-407e-ac4b-22e1736966ac" (UID: "2d9edca1-e32c-407e-ac4b-22e1736966ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:16:56.240994 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:56.240973    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9edca1-e32c-407e-ac4b-22e1736966ac-kube-api-access-6dplt" pod "2d9edca1-e32c-407e-ac4b-22e1736966ac" (UID: "2d9edca1-e32c-407e-ac4b-22e1736966ac"). InnerVolumeSpecName "kube-api-access-6dplt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:16:56.263555 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:56.263533    4889 scope.go:122] "RemoveContainer" containerID="d924930151d9565670cb7ce088cf3ee8dc3ba73f55d92484fc7f7c34a0a9c107"
Oct 30 23:16:56.263915 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:16:56.263889    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d924930151d9565670cb7ce088cf3ee8dc3ba73f55d92484fc7f7c34a0a9c107\": container with ID starting with d924930151d9565670cb7ce088cf3ee8dc3ba73f55d92484fc7f7c34a0a9c107 not found: ID does not exist" containerID="d924930151d9565670cb7ce088cf3ee8dc3ba73f55d92484fc7f7c34a0a9c107"
Oct 30 23:16:56.263969 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:56.263927    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d924930151d9565670cb7ce088cf3ee8dc3ba73f55d92484fc7f7c34a0a9c107"} err="failed to get container status \"d924930151d9565670cb7ce088cf3ee8dc3ba73f55d92484fc7f7c34a0a9c107\": rpc error: code = NotFound desc = could not find container \"d924930151d9565670cb7ce088cf3ee8dc3ba73f55d92484fc7f7c34a0a9c107\": container with ID starting with d924930151d9565670cb7ce088cf3ee8dc3ba73f55d92484fc7f7c34a0a9c107 not found: ID does not exist"
Oct 30 23:16:56.338370 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:56.338337    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d9edca1-e32c-407e-ac4b-22e1736966ac-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:16:56.338370 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:56.338365    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dplt\" (UniqueName: \"kubernetes.io/projected/2d9edca1-e32c-407e-ac4b-22e1736966ac-kube-api-access-6dplt\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:16:56.338370 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:56.338377    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9edca1-e32c-407e-ac4b-22e1736966ac-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:16:56.338624 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:56.338386    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d9edca1-e32c-407e-ac4b-22e1736966ac-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:16:56.583558 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:56.583531    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw"]
Oct 30 23:16:56.585604 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:56.585583    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef9544-59db788b7f-tkkgw"]
Oct 30 23:16:56.773519 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:56.773491    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d9edca1-e32c-407e-ac4b-22e1736966ac" path="/var/lib/kubelet/pods/2d9edca1-e32c-407e-ac4b-22e1736966ac/volumes"
Oct 30 23:16:58.391047 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:58.391003    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9" podUID="67976ad6-64a0-43aa-8167-dad668fdcb4e" containerName="payload-processing" containerID="cri-o://4bfbac3263973641e17af7541624946e70557e72ba2a9c7b4eaa4198f383bb0b" gracePeriod=25
Oct 30 23:16:58.422879 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:16:58.422825    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67" podUID="0650ad04-1155-4626-8480-53e7e3d82e39" containerName="payload-pre-processing" containerID="cri-o://4496e9250db477a7d979cf80b791329794f5bd859817037e87b24088ac4c8082" gracePeriod=25
Oct 30 23:17:02.364217 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:02.364140    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:17:02.556174 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:02.556137    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:17:02.630190 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:02.630081    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"]
Oct 30 23:17:02.630461 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:02.630394    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l" podUID="258a920d-b0b1-4eae-9437-4289516d310a" containerName="istio-proxy" containerID="cri-o://1cca7a060eab06d9ce5fa506b2daf12bca7a83f6079aed10b6bbcb72ff186f5e" gracePeriod=30
Oct 30 23:17:02.727735 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:02.727692    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z" podUID="163e6247-764e-4fcb-af26-f8896c00da5a" containerName="main" containerID="cri-o://009c8ffdb2452b01be97d38db4c08ce180581d5fcd88cf1ddd4c9b91830a376d" gracePeriod=45
Oct 30 23:17:02.732752 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:02.732712    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:17:02.993805 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:02.993780    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"
Oct 30 23:17:03.108665 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.108635    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-tmp-dir\") pod \"163e6247-764e-4fcb-af26-f8896c00da5a\" (UID: \"163e6247-764e-4fcb-af26-f8896c00da5a\") "
Oct 30 23:17:03.108879 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.108750    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-dshm\") pod \"163e6247-764e-4fcb-af26-f8896c00da5a\" (UID: \"163e6247-764e-4fcb-af26-f8896c00da5a\") "
Oct 30 23:17:03.108879 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.108810    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-home\" (UniqueName: \"kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-home\") pod \"163e6247-764e-4fcb-af26-f8896c00da5a\" (UID: \"163e6247-764e-4fcb-af26-f8896c00da5a\") "
Oct 30 23:17:03.108879 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.108852    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/163e6247-764e-4fcb-af26-f8896c00da5a-tls-certs\" (UniqueName: \"kubernetes.io/secret/163e6247-764e-4fcb-af26-f8896c00da5a-tls-certs\") pod \"163e6247-764e-4fcb-af26-f8896c00da5a\" (UID: \"163e6247-764e-4fcb-af26-f8896c00da5a\") "
Oct 30 23:17:03.108879 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.108880    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-model-cache\") pod \"163e6247-764e-4fcb-af26-f8896c00da5a\" (UID: \"163e6247-764e-4fcb-af26-f8896c00da5a\") "
Oct 30 23:17:03.109126 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.108906    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/163e6247-764e-4fcb-af26-f8896c00da5a-kube-api-access-r48sm\" (UniqueName: \"kubernetes.io/projected/163e6247-764e-4fcb-af26-f8896c00da5a-kube-api-access-r48sm\") pod \"163e6247-764e-4fcb-af26-f8896c00da5a\" (UID: \"163e6247-764e-4fcb-af26-f8896c00da5a\") "
Oct 30 23:17:03.109126 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.108946    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-tmp-dir" pod "163e6247-764e-4fcb-af26-f8896c00da5a" (UID: "163e6247-764e-4fcb-af26-f8896c00da5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:17:03.109126 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.109072    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-home" pod "163e6247-764e-4fcb-af26-f8896c00da5a" (UID: "163e6247-764e-4fcb-af26-f8896c00da5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:17:03.109325 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.109192    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-model-cache" pod "163e6247-764e-4fcb-af26-f8896c00da5a" (UID: "163e6247-764e-4fcb-af26-f8896c00da5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:17:03.109325 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.109307    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:17:03.109603 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.109325    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:17:03.109603 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.109341    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:17:03.111216 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.111185    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-dshm" pod "163e6247-764e-4fcb-af26-f8896c00da5a" (UID: "163e6247-764e-4fcb-af26-f8896c00da5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:17:03.111410 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.111338    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/163e6247-764e-4fcb-af26-f8896c00da5a-tls-certs" pod "163e6247-764e-4fcb-af26-f8896c00da5a" (UID: "163e6247-764e-4fcb-af26-f8896c00da5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:17:03.111410 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.111398    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/163e6247-764e-4fcb-af26-f8896c00da5a-kube-api-access-r48sm" pod "163e6247-764e-4fcb-af26-f8896c00da5a" (UID: "163e6247-764e-4fcb-af26-f8896c00da5a"). InnerVolumeSpecName "kube-api-access-r48sm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:17:03.210447 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.210371    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/163e6247-764e-4fcb-af26-f8896c00da5a-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:17:03.210447 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.210401    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/163e6247-764e-4fcb-af26-f8896c00da5a-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:17:03.210447 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.210416    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r48sm\" (UniqueName: \"kubernetes.io/projected/163e6247-764e-4fcb-af26-f8896c00da5a-kube-api-access-r48sm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:17:03.284606 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.284571    4889 generic.go:356] "Generic (PLEG): container finished" podID="163e6247-764e-4fcb-af26-f8896c00da5a" containerID="009c8ffdb2452b01be97d38db4c08ce180581d5fcd88cf1ddd4c9b91830a376d" exitCode=0
Oct 30 23:17:03.284800 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.284623    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"
Oct 30 23:17:03.284800 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.284666    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z" event={"ID":"163e6247-764e-4fcb-af26-f8896c00da5a","Type":"ContainerDied","Data":"009c8ffdb2452b01be97d38db4c08ce180581d5fcd88cf1ddd4c9b91830a376d"}
Oct 30 23:17:03.284800 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.284713    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z" event={"ID":"163e6247-764e-4fcb-af26-f8896c00da5a","Type":"ContainerDied","Data":"074b25921200160802bdd8c4131a264388a888030d6c419fe98dff999294a691"}
Oct 30 23:17:03.284800 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.284730    4889 scope.go:122] "RemoveContainer" containerID="009c8ffdb2452b01be97d38db4c08ce180581d5fcd88cf1ddd4c9b91830a376d"
Oct 30 23:17:03.294544 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.294528    4889 scope.go:122] "RemoveContainer" containerID="009c8ffdb2452b01be97d38db4c08ce180581d5fcd88cf1ddd4c9b91830a376d"
Oct 30 23:17:03.294782 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:17:03.294767    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"009c8ffdb2452b01be97d38db4c08ce180581d5fcd88cf1ddd4c9b91830a376d\": container with ID starting with 009c8ffdb2452b01be97d38db4c08ce180581d5fcd88cf1ddd4c9b91830a376d not found: ID does not exist" containerID="009c8ffdb2452b01be97d38db4c08ce180581d5fcd88cf1ddd4c9b91830a376d"
Oct 30 23:17:03.294829 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.294792    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"009c8ffdb2452b01be97d38db4c08ce180581d5fcd88cf1ddd4c9b91830a376d"} err="failed to get container status \"009c8ffdb2452b01be97d38db4c08ce180581d5fcd88cf1ddd4c9b91830a376d\": rpc error: code = NotFound desc = could not find container \"009c8ffdb2452b01be97d38db4c08ce180581d5fcd88cf1ddd4c9b91830a376d\": container with ID starting with 009c8ffdb2452b01be97d38db4c08ce180581d5fcd88cf1ddd4c9b91830a376d not found: ID does not exist"
Oct 30 23:17:03.340711 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.340683    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"]
Oct 30 23:17:03.347373 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.347353    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-ef9544/simulator-single-main-ef9544-kserve-6bcd56b79-qlq8z"]
Oct 30 23:17:03.716661 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.716617    4889 patch_prober.go:29] interesting pod/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:17:03.717062 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:03.716686    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l" podUID="258a920d-b0b1-4eae-9437-4289516d310a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:17:04.773358 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:04.773323    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="163e6247-764e-4fcb-af26-f8896c00da5a" path="/var/lib/kubelet/pods/163e6247-764e-4fcb-af26-f8896c00da5a/volumes"
Oct 30 23:17:06.859471 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:06.859440    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9" podUID="67976ad6-64a0-43aa-8167-dad668fdcb4e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:17:06.859855 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:06.859439    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67" podUID="0650ad04-1155-4626-8480-53e7e3d82e39" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:17:07.894666 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:07.894642    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:17:08.062713 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.062672    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-credential-socket\") pod \"258a920d-b0b1-4eae-9437-4289516d310a\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") "
Oct 30 23:17:08.062713 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.062710    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/258a920d-b0b1-4eae-9437-4289516d310a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/258a920d-b0b1-4eae-9437-4289516d310a-istiod-ca-cert\") pod \"258a920d-b0b1-4eae-9437-4289516d310a\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") "
Oct 30 23:17:08.062999 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.062735    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/258a920d-b0b1-4eae-9437-4289516d310a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/258a920d-b0b1-4eae-9437-4289516d310a-istio-podinfo\") pod \"258a920d-b0b1-4eae-9437-4289516d310a\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") "
Oct 30 23:17:08.062999 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.062760    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/258a920d-b0b1-4eae-9437-4289516d310a-istio-token\" (UniqueName: \"kubernetes.io/projected/258a920d-b0b1-4eae-9437-4289516d310a-istio-token\") pod \"258a920d-b0b1-4eae-9437-4289516d310a\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") "
Oct 30 23:17:08.062999 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.062795    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-istio-envoy\") pod \"258a920d-b0b1-4eae-9437-4289516d310a\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") "
Oct 30 23:17:08.062999 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.062859    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-workload-certs\") pod \"258a920d-b0b1-4eae-9437-4289516d310a\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") "
Oct 30 23:17:08.062999 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.062938    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/258a920d-b0b1-4eae-9437-4289516d310a-kube-api-access-9rg96\" (UniqueName: \"kubernetes.io/projected/258a920d-b0b1-4eae-9437-4289516d310a-kube-api-access-9rg96\") pod \"258a920d-b0b1-4eae-9437-4289516d310a\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") "
Oct 30 23:17:08.062999 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.062971    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-workload-socket\") pod \"258a920d-b0b1-4eae-9437-4289516d310a\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") "
Oct 30 23:17:08.063387 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.063008    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-istio-data\") pod \"258a920d-b0b1-4eae-9437-4289516d310a\" (UID: \"258a920d-b0b1-4eae-9437-4289516d310a\") "
Oct 30 23:17:08.063387 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.063092    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-credential-socket" pod "258a920d-b0b1-4eae-9437-4289516d310a" (UID: "258a920d-b0b1-4eae-9437-4289516d310a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:17:08.063387 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.063336    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-workload-certs" pod "258a920d-b0b1-4eae-9437-4289516d310a" (UID: "258a920d-b0b1-4eae-9437-4289516d310a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:17:08.063387 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.063352    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-workload-socket" pod "258a920d-b0b1-4eae-9437-4289516d310a" (UID: "258a920d-b0b1-4eae-9437-4289516d310a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:17:08.063781 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.063459    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/258a920d-b0b1-4eae-9437-4289516d310a-istiod-ca-cert" pod "258a920d-b0b1-4eae-9437-4289516d310a" (UID: "258a920d-b0b1-4eae-9437-4289516d310a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:17:08.063781 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.063525    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:17:08.063781 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.063544    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:17:08.063781 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.063553    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:17:08.063781 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.063555    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-istio-data" pod "258a920d-b0b1-4eae-9437-4289516d310a" (UID: "258a920d-b0b1-4eae-9437-4289516d310a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:17:08.065391 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.065364    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/258a920d-b0b1-4eae-9437-4289516d310a-istio-podinfo" pod "258a920d-b0b1-4eae-9437-4289516d310a" (UID: "258a920d-b0b1-4eae-9437-4289516d310a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:17:08.065391 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.065376    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/258a920d-b0b1-4eae-9437-4289516d310a-istio-token" pod "258a920d-b0b1-4eae-9437-4289516d310a" (UID: "258a920d-b0b1-4eae-9437-4289516d310a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:17:08.065542 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.065501    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-istio-envoy" pod "258a920d-b0b1-4eae-9437-4289516d310a" (UID: "258a920d-b0b1-4eae-9437-4289516d310a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:17:08.065804 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.065782    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/258a920d-b0b1-4eae-9437-4289516d310a-kube-api-access-9rg96" pod "258a920d-b0b1-4eae-9437-4289516d310a" (UID: "258a920d-b0b1-4eae-9437-4289516d310a"). InnerVolumeSpecName "kube-api-access-9rg96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:17:08.164898 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.164861    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/258a920d-b0b1-4eae-9437-4289516d310a-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:17:08.164898 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.164893    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/258a920d-b0b1-4eae-9437-4289516d310a-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:17:08.165114 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.164908    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/258a920d-b0b1-4eae-9437-4289516d310a-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:17:08.165114 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.164918    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:17:08.165114 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.164926    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rg96\" (UniqueName: \"kubernetes.io/projected/258a920d-b0b1-4eae-9437-4289516d310a-kube-api-access-9rg96\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:17:08.165114 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.164934    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/258a920d-b0b1-4eae-9437-4289516d310a-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:17:08.312621 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.312586    4889 generic.go:356] "Generic (PLEG): container finished" podID="258a920d-b0b1-4eae-9437-4289516d310a" containerID="1cca7a060eab06d9ce5fa506b2daf12bca7a83f6079aed10b6bbcb72ff186f5e" exitCode=0
Oct 30 23:17:08.312865 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.312652    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l" event={"ID":"258a920d-b0b1-4eae-9437-4289516d310a","Type":"ContainerDied","Data":"1cca7a060eab06d9ce5fa506b2daf12bca7a83f6079aed10b6bbcb72ff186f5e"}
Oct 30 23:17:08.312865 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.312656    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"
Oct 30 23:17:08.312865 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.312693    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l" event={"ID":"258a920d-b0b1-4eae-9437-4289516d310a","Type":"ContainerDied","Data":"9df6c248c8f8e69ac7268565f3a939c9c02070c821b3b8f937851c821c9d8d8b"}
Oct 30 23:17:08.312865 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.312716    4889 scope.go:122] "RemoveContainer" containerID="1cca7a060eab06d9ce5fa506b2daf12bca7a83f6079aed10b6bbcb72ff186f5e"
Oct 30 23:17:08.322795 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.322777    4889 scope.go:122] "RemoveContainer" containerID="54c4651de56eeb3868e6462aba6a08d4d99103616ae545c7eea2e7cb55645ead"
Oct 30 23:17:08.331730 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.331707    4889 scope.go:122] "RemoveContainer" containerID="1cca7a060eab06d9ce5fa506b2daf12bca7a83f6079aed10b6bbcb72ff186f5e"
Oct 30 23:17:08.331977 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:17:08.331955    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cca7a060eab06d9ce5fa506b2daf12bca7a83f6079aed10b6bbcb72ff186f5e\": container with ID starting with 1cca7a060eab06d9ce5fa506b2daf12bca7a83f6079aed10b6bbcb72ff186f5e not found: ID does not exist" containerID="1cca7a060eab06d9ce5fa506b2daf12bca7a83f6079aed10b6bbcb72ff186f5e"
Oct 30 23:17:08.332044 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.331986    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cca7a060eab06d9ce5fa506b2daf12bca7a83f6079aed10b6bbcb72ff186f5e"} err="failed to get container status \"1cca7a060eab06d9ce5fa506b2daf12bca7a83f6079aed10b6bbcb72ff186f5e\": rpc error: code = NotFound desc = could not find container \"1cca7a060eab06d9ce5fa506b2daf12bca7a83f6079aed10b6bbcb72ff186f5e\": container with ID starting with 1cca7a060eab06d9ce5fa506b2daf12bca7a83f6079aed10b6bbcb72ff186f5e not found: ID does not exist"
Oct 30 23:17:08.332044 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.332002    4889 scope.go:122] "RemoveContainer" containerID="54c4651de56eeb3868e6462aba6a08d4d99103616ae545c7eea2e7cb55645ead"
Oct 30 23:17:08.332262 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:17:08.332243    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54c4651de56eeb3868e6462aba6a08d4d99103616ae545c7eea2e7cb55645ead\": container with ID starting with 54c4651de56eeb3868e6462aba6a08d4d99103616ae545c7eea2e7cb55645ead not found: ID does not exist" containerID="54c4651de56eeb3868e6462aba6a08d4d99103616ae545c7eea2e7cb55645ead"
Oct 30 23:17:08.332355 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.332269    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54c4651de56eeb3868e6462aba6a08d4d99103616ae545c7eea2e7cb55645ead"} err="failed to get container status \"54c4651de56eeb3868e6462aba6a08d4d99103616ae545c7eea2e7cb55645ead\": rpc error: code = NotFound desc = could not find container \"54c4651de56eeb3868e6462aba6a08d4d99103616ae545c7eea2e7cb55645ead\": container with ID starting with 54c4651de56eeb3868e6462aba6a08d4d99103616ae545c7eea2e7cb55645ead not found: ID does not exist"
Oct 30 23:17:08.375766 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.375739    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"]
Oct 30 23:17:08.378329 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.378305    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ef9544-openshift-default-68f86596d5hj74l"]
Oct 30 23:17:08.773526 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:08.773498    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="258a920d-b0b1-4eae-9437-4289516d310a" path="/var/lib/kubelet/pods/258a920d-b0b1-4eae-9437-4289516d310a/volumes"
Oct 30 23:17:09.892292 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:09.892251    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"]
Oct 30 23:17:09.892868 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:09.892847    4889 cpu_manager.go:414] "containerMap: removing container" podUID="2d9edca1-e32c-407e-ac4b-22e1736966ac" containerName="maas-api"
Oct 30 23:17:09.892969 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:09.892870    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d9edca1-e32c-407e-ac4b-22e1736966ac" containerName="maas-api"
Oct 30 23:17:09.892969 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:09.892901    4889 cpu_manager.go:414] "containerMap: removing container" podUID="163e6247-764e-4fcb-af26-f8896c00da5a" containerName="main"
Oct 30 23:17:09.892969 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:09.892909    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="163e6247-764e-4fcb-af26-f8896c00da5a" containerName="main"
Oct 30 23:17:09.892969 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:09.892927    4889 cpu_manager.go:414] "containerMap: removing container" podUID="258a920d-b0b1-4eae-9437-4289516d310a" containerName="istio-proxy"
Oct 30 23:17:09.892969 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:09.892934    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="258a920d-b0b1-4eae-9437-4289516d310a" containerName="istio-proxy"
Oct 30 23:17:09.892969 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:09.892970    4889 cpu_manager.go:414] "containerMap: removing container" podUID="258a920d-b0b1-4eae-9437-4289516d310a" containerName="istio-proxy"
Oct 30 23:17:09.893305 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:09.892978    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="258a920d-b0b1-4eae-9437-4289516d310a" containerName="istio-proxy"
Oct 30 23:17:09.893305 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:09.893082    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="258a920d-b0b1-4eae-9437-4289516d310a" containerName="istio-proxy"
Oct 30 23:17:09.893305 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:09.893105    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d9edca1-e32c-407e-ac4b-22e1736966ac" containerName="maas-api"
Oct 30 23:17:09.893305 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:09.893117    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="163e6247-764e-4fcb-af26-f8896c00da5a" containerName="main"
Oct 30 23:17:09.897806 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:09.897783    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"
Oct 30 23:17:09.914417 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:09.914388    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"]
Oct 30 23:17:09.925885 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:09.925859    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-2917e5f5-kserve-self-signed-certs\""
Oct 30 23:17:09.981714 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:09.981680    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-model-cache\") pod \"e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7\" (UID: \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\") " pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"
Oct 30 23:17:09.981907 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:09.981813    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5722j\" (UniqueName: \"kubernetes.io/projected/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-kube-api-access-5722j\") pod \"e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7\" (UID: \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\") " pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"
Oct 30 23:17:09.981907 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:09.981858    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-home\") pod \"e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7\" (UID: \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\") " pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"
Oct 30 23:17:09.982021 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:09.981938    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-tls-certs\") pod \"e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7\" (UID: \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\") " pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"
Oct 30 23:17:09.982064 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:09.982041    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-dshm\") pod \"e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7\" (UID: \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\") " pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"
Oct 30 23:17:09.982107 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:09.982078    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-tmp-dir\") pod \"e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7\" (UID: \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\") " pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"
Oct 30 23:17:10.032400 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:10.032363    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:17:10.082874 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:10.082826    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5722j\" (UniqueName: \"kubernetes.io/projected/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-kube-api-access-5722j\") pod \"e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7\" (UID: \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\") " pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"
Oct 30 23:17:10.082874 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:10.082879    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-home\") pod \"e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7\" (UID: \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\") " pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"
Oct 30 23:17:10.083169 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:10.082916    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-tls-certs\") pod \"e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7\" (UID: \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\") " pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"
Oct 30 23:17:10.083169 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:10.082977    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-dshm\") pod \"e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7\" (UID: \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\") " pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"
Oct 30 23:17:10.083169 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:10.082995    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-tmp-dir\") pod \"e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7\" (UID: \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\") " pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"
Oct 30 23:17:10.083169 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:10.083029    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-model-cache\") pod \"e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7\" (UID: \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\") " pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"
Oct 30 23:17:10.083691 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:10.083508    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-model-cache\") pod \"e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7\" (UID: \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\") " pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"
Oct 30 23:17:10.084400 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:10.084379    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-home\") pod \"e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7\" (UID: \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\") " pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"
Oct 30 23:17:10.085211 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:10.085138    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-tmp-dir\") pod \"e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7\" (UID: \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\") " pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"
Oct 30 23:17:10.088273 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:10.088246    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-dshm\") pod \"e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7\" (UID: \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\") " pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"
Oct 30 23:17:10.088416 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:10.088384    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-tls-certs\") pod \"e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7\" (UID: \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\") " pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"
Oct 30 23:17:10.105473 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:10.105443    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5722j\" (UniqueName: \"kubernetes.io/projected/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-kube-api-access-5722j\") pod \"e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7\" (UID: \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\") " pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"
Oct 30 23:17:10.211892 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:10.211808    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"
Oct 30 23:17:10.304904 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:17:10.304812    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:17:10.512084 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:10.512055    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"]
Oct 30 23:17:10.513189 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:17:10.513163    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9fc8cad_c0ac_4b7d_a716_9bb3319f7f8f.slice/crio-698ae50120f2a7c4308e380393b49f97fd5ff6798205515efccb384a3293618a WatchSource:0}: Error finding container 698ae50120f2a7c4308e380393b49f97fd5ff6798205515efccb384a3293618a: Status 404 returned error can't find the container with id 698ae50120f2a7c4308e380393b49f97fd5ff6798205515efccb384a3293618a
Oct 30 23:17:11.330177 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:11.330146    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7" event={"ID":"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f","Type":"ContainerStarted","Data":"72a0b9104307a0b84e78056f57aef5a6b7a26098558e996fcbdcd4763b3ee1ce"}
Oct 30 23:17:11.330593 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:11.330185    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7" event={"ID":"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f","Type":"ContainerStarted","Data":"698ae50120f2a7c4308e380393b49f97fd5ff6798205515efccb384a3293618a"}
Oct 30 23:17:11.376587 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:11.376519    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7" podStartSLOduration=2.376505925 podStartE2EDuration="2.376505925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:17:11.374612051 +0000 UTC m=+4216.730978366" watchObservedRunningTime="2026-10-30 23:17:11.376505925 +0000 UTC m=+4216.732872240"
Oct 30 23:17:12.211949 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:12.211918    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"
Oct 30 23:17:15.987481 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:15.987441    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:17:16.858759 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:16.858713    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9" podUID="67976ad6-64a0-43aa-8167-dad668fdcb4e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:17:16.858953 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:16.858713    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67" podUID="0650ad04-1155-4626-8480-53e7e3d82e39" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:17:16.903359 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:16.903322    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:17:18.683204 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:18.683178    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9"
Oct 30 23:17:18.707469 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:18.707445    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67"
Oct 30 23:17:18.727155 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:18.727128    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0650ad04-1155-4626-8480-53e7e3d82e39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0650ad04-1155-4626-8480-53e7e3d82e39-plugins-config-volume\") pod \"0650ad04-1155-4626-8480-53e7e3d82e39\" (UID: \"0650ad04-1155-4626-8480-53e7e3d82e39\") "
Oct 30 23:17:18.727320 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:18.727187    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0650ad04-1155-4626-8480-53e7e3d82e39-kube-api-access-h4mqf\" (UniqueName: \"kubernetes.io/projected/0650ad04-1155-4626-8480-53e7e3d82e39-kube-api-access-h4mqf\") pod \"0650ad04-1155-4626-8480-53e7e3d82e39\" (UID: \"0650ad04-1155-4626-8480-53e7e3d82e39\") "
Oct 30 23:17:18.727320 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:18.727264    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67976ad6-64a0-43aa-8167-dad668fdcb4e-kube-api-access-jts5x\" (UniqueName: \"kubernetes.io/projected/67976ad6-64a0-43aa-8167-dad668fdcb4e-kube-api-access-jts5x\") pod \"67976ad6-64a0-43aa-8167-dad668fdcb4e\" (UID: \"67976ad6-64a0-43aa-8167-dad668fdcb4e\") "
Oct 30 23:17:18.727320 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:18.727296    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67976ad6-64a0-43aa-8167-dad668fdcb4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67976ad6-64a0-43aa-8167-dad668fdcb4e-plugins-config-volume\") pod \"67976ad6-64a0-43aa-8167-dad668fdcb4e\" (UID: \"67976ad6-64a0-43aa-8167-dad668fdcb4e\") "
Oct 30 23:17:18.727525 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:18.727503    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0650ad04-1155-4626-8480-53e7e3d82e39-plugins-config-volume" pod "0650ad04-1155-4626-8480-53e7e3d82e39" (UID: "0650ad04-1155-4626-8480-53e7e3d82e39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:17:18.727932 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:18.727629    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0650ad04-1155-4626-8480-53e7e3d82e39-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:17:18.728207 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:18.728065    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67976ad6-64a0-43aa-8167-dad668fdcb4e-plugins-config-volume" pod "67976ad6-64a0-43aa-8167-dad668fdcb4e" (UID: "67976ad6-64a0-43aa-8167-dad668fdcb4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:17:18.729825 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:18.729801    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0650ad04-1155-4626-8480-53e7e3d82e39-kube-api-access-h4mqf" pod "0650ad04-1155-4626-8480-53e7e3d82e39" (UID: "0650ad04-1155-4626-8480-53e7e3d82e39"). InnerVolumeSpecName "kube-api-access-h4mqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:17:18.730169 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:18.730146    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67976ad6-64a0-43aa-8167-dad668fdcb4e-kube-api-access-jts5x" pod "67976ad6-64a0-43aa-8167-dad668fdcb4e" (UID: "67976ad6-64a0-43aa-8167-dad668fdcb4e"). InnerVolumeSpecName "kube-api-access-jts5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:17:18.828681 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:18.828650    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4mqf\" (UniqueName: \"kubernetes.io/projected/0650ad04-1155-4626-8480-53e7e3d82e39-kube-api-access-h4mqf\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:17:18.828681 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:18.828678    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jts5x\" (UniqueName: \"kubernetes.io/projected/67976ad6-64a0-43aa-8167-dad668fdcb4e-kube-api-access-jts5x\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:17:18.828681 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:18.828687    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67976ad6-64a0-43aa-8167-dad668fdcb4e-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:17:19.422275 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:19.422208    4889 generic.go:356] "Generic (PLEG): container finished" podID="0650ad04-1155-4626-8480-53e7e3d82e39" containerID="4496e9250db477a7d979cf80b791329794f5bd859817037e87b24088ac4c8082" exitCode=0
Oct 30 23:17:19.422451 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:19.422282    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67"
Oct 30 23:17:19.422451 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:19.422280    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67" event={"ID":"0650ad04-1155-4626-8480-53e7e3d82e39","Type":"ContainerDied","Data":"4496e9250db477a7d979cf80b791329794f5bd859817037e87b24088ac4c8082"}
Oct 30 23:17:19.422567 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:19.422459    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67" event={"ID":"0650ad04-1155-4626-8480-53e7e3d82e39","Type":"ContainerDied","Data":"2f83b7882d8e1724f53b8296423211d74600a09b80694264b1c822b11ec453c5"}
Oct 30 23:17:19.422567 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:19.422508    4889 scope.go:122] "RemoveContainer" containerID="4496e9250db477a7d979cf80b791329794f5bd859817037e87b24088ac4c8082"
Oct 30 23:17:19.423861 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:19.423839    4889 generic.go:356] "Generic (PLEG): container finished" podID="67976ad6-64a0-43aa-8167-dad668fdcb4e" containerID="4bfbac3263973641e17af7541624946e70557e72ba2a9c7b4eaa4198f383bb0b" exitCode=0
Oct 30 23:17:19.423983 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:19.423880    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9"
Oct 30 23:17:19.423983 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:19.423924    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9" event={"ID":"67976ad6-64a0-43aa-8167-dad668fdcb4e","Type":"ContainerDied","Data":"4bfbac3263973641e17af7541624946e70557e72ba2a9c7b4eaa4198f383bb0b"}
Oct 30 23:17:19.423983 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:19.423961    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9" event={"ID":"67976ad6-64a0-43aa-8167-dad668fdcb4e","Type":"ContainerDied","Data":"a9260343ccd3bc10bf1aa1ba674e8ebc9d0b28fb36e6cc4b5faeace8ed23559f"}
Oct 30 23:17:19.431806 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:19.431776    4889 scope.go:122] "RemoveContainer" containerID="4496e9250db477a7d979cf80b791329794f5bd859817037e87b24088ac4c8082"
Oct 30 23:17:19.432065 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:17:19.432046    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4496e9250db477a7d979cf80b791329794f5bd859817037e87b24088ac4c8082\": container with ID starting with 4496e9250db477a7d979cf80b791329794f5bd859817037e87b24088ac4c8082 not found: ID does not exist" containerID="4496e9250db477a7d979cf80b791329794f5bd859817037e87b24088ac4c8082"
Oct 30 23:17:19.432131 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:19.432076    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4496e9250db477a7d979cf80b791329794f5bd859817037e87b24088ac4c8082"} err="failed to get container status \"4496e9250db477a7d979cf80b791329794f5bd859817037e87b24088ac4c8082\": rpc error: code = NotFound desc = could not find container \"4496e9250db477a7d979cf80b791329794f5bd859817037e87b24088ac4c8082\": container with ID starting with 4496e9250db477a7d979cf80b791329794f5bd859817037e87b24088ac4c8082 not found: ID does not exist"
Oct 30 23:17:19.432131 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:19.432095    4889 scope.go:122] "RemoveContainer" containerID="4bfbac3263973641e17af7541624946e70557e72ba2a9c7b4eaa4198f383bb0b"
Oct 30 23:17:19.440118 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:19.440103    4889 scope.go:122] "RemoveContainer" containerID="4bfbac3263973641e17af7541624946e70557e72ba2a9c7b4eaa4198f383bb0b"
Oct 30 23:17:19.440444 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:17:19.440422    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bfbac3263973641e17af7541624946e70557e72ba2a9c7b4eaa4198f383bb0b\": container with ID starting with 4bfbac3263973641e17af7541624946e70557e72ba2a9c7b4eaa4198f383bb0b not found: ID does not exist" containerID="4bfbac3263973641e17af7541624946e70557e72ba2a9c7b4eaa4198f383bb0b"
Oct 30 23:17:19.440547 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:19.440450    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bfbac3263973641e17af7541624946e70557e72ba2a9c7b4eaa4198f383bb0b"} err="failed to get container status \"4bfbac3263973641e17af7541624946e70557e72ba2a9c7b4eaa4198f383bb0b\": rpc error: code = NotFound desc = could not find container \"4bfbac3263973641e17af7541624946e70557e72ba2a9c7b4eaa4198f383bb0b\": container with ID starting with 4bfbac3263973641e17af7541624946e70557e72ba2a9c7b4eaa4198f383bb0b not found: ID does not exist"
Oct 30 23:17:19.477397 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:19.477366    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9"]
Oct 30 23:17:19.491268 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:19.491243    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ef9544-6849c57fcsxzg9"]
Oct 30 23:17:19.527895 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:19.527865    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67"]
Oct 30 23:17:19.531313 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:19.531288    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef9544-69bf6pfp67"]
Oct 30 23:17:20.212491 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:20.212442    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"
Oct 30 23:17:20.228218 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:20.228190    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"
Oct 30 23:17:20.441949 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:20.441917    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"
Oct 30 23:17:20.772548 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:20.772516    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0650ad04-1155-4626-8480-53e7e3d82e39" path="/var/lib/kubelet/pods/0650ad04-1155-4626-8480-53e7e3d82e39/volumes"
Oct 30 23:17:20.772916 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:20.772902    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67976ad6-64a0-43aa-8167-dad668fdcb4e" path="/var/lib/kubelet/pods/67976ad6-64a0-43aa-8167-dad668fdcb4e/volumes"
Oct 30 23:17:22.573252 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:22.573192    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:17:29.209891 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.209854    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"]
Oct 30 23:17:29.210446 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.210271    4889 cpu_manager.go:414] "containerMap: removing container" podUID="0650ad04-1155-4626-8480-53e7e3d82e39" containerName="payload-pre-processing"
Oct 30 23:17:29.210446 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.210285    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0650ad04-1155-4626-8480-53e7e3d82e39" containerName="payload-pre-processing"
Oct 30 23:17:29.210446 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.210318    4889 cpu_manager.go:414] "containerMap: removing container" podUID="67976ad6-64a0-43aa-8167-dad668fdcb4e" containerName="payload-processing"
Oct 30 23:17:29.210446 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.210323    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67976ad6-64a0-43aa-8167-dad668fdcb4e" containerName="payload-processing"
Oct 30 23:17:29.210446 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.210378    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="67976ad6-64a0-43aa-8167-dad668fdcb4e" containerName="payload-processing"
Oct 30 23:17:29.210446 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.210394    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="0650ad04-1155-4626-8480-53e7e3d82e39" containerName="payload-pre-processing"
Oct 30 23:17:29.210446 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.210406    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="258a920d-b0b1-4eae-9437-4289516d310a" containerName="istio-proxy"
Oct 30 23:17:29.215248 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.215213    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"
Oct 30 23:17:29.230669 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.230635    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"]
Oct 30 23:17:29.242151 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.242119    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-2917e5f5-kserve-self-signed-certs\""
Oct 30 23:17:29.331541 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.331498    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:17:29.334574 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.334551    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-home\") pod \"e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd\" (UID: \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\") " pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"
Oct 30 23:17:29.334691 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.334582    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-model-cache\") pod \"e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd\" (UID: \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\") " pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"
Oct 30 23:17:29.334740 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.334701    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7klff\" (UniqueName: \"kubernetes.io/projected/75d65a47-3e5b-4099-890c-f7b5cbdb5654-kube-api-access-7klff\") pod \"e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd\" (UID: \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\") " pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"
Oct 30 23:17:29.334782 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.334769    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-tmp-dir\") pod \"e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd\" (UID: \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\") " pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"
Oct 30 23:17:29.334836 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.334823    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75d65a47-3e5b-4099-890c-f7b5cbdb5654-tls-certs\") pod \"e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd\" (UID: \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\") " pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"
Oct 30 23:17:29.334880 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.334868    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-dshm\") pod \"e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd\" (UID: \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\") " pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"
Oct 30 23:17:29.435778 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.435740    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7klff\" (UniqueName: \"kubernetes.io/projected/75d65a47-3e5b-4099-890c-f7b5cbdb5654-kube-api-access-7klff\") pod \"e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd\" (UID: \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\") " pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"
Oct 30 23:17:29.435991 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.435833    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-tmp-dir\") pod \"e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd\" (UID: \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\") " pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"
Oct 30 23:17:29.435991 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.435869    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75d65a47-3e5b-4099-890c-f7b5cbdb5654-tls-certs\") pod \"e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd\" (UID: \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\") " pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"
Oct 30 23:17:29.435991 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.435908    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-dshm\") pod \"e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd\" (UID: \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\") " pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"
Oct 30 23:17:29.435991 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.435938    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-home\") pod \"e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd\" (UID: \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\") " pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"
Oct 30 23:17:29.435991 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.435961    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-model-cache\") pod \"e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd\" (UID: \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\") " pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"
Oct 30 23:17:29.436536 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.436501    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-model-cache\") pod \"e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd\" (UID: \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\") " pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"
Oct 30 23:17:29.436756 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.436724    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-tmp-dir\") pod \"e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd\" (UID: \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\") " pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"
Oct 30 23:17:29.436997 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.436916    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-home\") pod \"e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd\" (UID: \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\") " pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"
Oct 30 23:17:29.439096 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.439065    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75d65a47-3e5b-4099-890c-f7b5cbdb5654-tls-certs\") pod \"e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd\" (UID: \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\") " pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"
Oct 30 23:17:29.439496 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.439470    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-dshm\") pod \"e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd\" (UID: \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\") " pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"
Oct 30 23:17:29.456313 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.456290    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7klff\" (UniqueName: \"kubernetes.io/projected/75d65a47-3e5b-4099-890c-f7b5cbdb5654-kube-api-access-7klff\") pod \"e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd\" (UID: \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\") " pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"
Oct 30 23:17:29.527876 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.527848    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"
Oct 30 23:17:29.828578 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:29.828552    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"]
Oct 30 23:17:29.829543 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:17:29.829517    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75d65a47_3e5b_4099_890c_f7b5cbdb5654.slice/crio-a569bcb56a8c118cc1ac31a0ec8ee39435e0e7f8cd0bc3eb866545889864af32 WatchSource:0}: Error finding container a569bcb56a8c118cc1ac31a0ec8ee39435e0e7f8cd0bc3eb866545889864af32: Status 404 returned error can't find the container with id a569bcb56a8c118cc1ac31a0ec8ee39435e0e7f8cd0bc3eb866545889864af32
Oct 30 23:17:30.481323 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:30.481287    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd" event={"ID":"75d65a47-3e5b-4099-890c-f7b5cbdb5654","Type":"ContainerStarted","Data":"b1e1c44b667909f515d705a0e23bdedb60096b5dd514b1a1615fdb84a51ea7bd"}
Oct 30 23:17:30.481733 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:30.481339    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd" event={"ID":"75d65a47-3e5b-4099-890c-f7b5cbdb5654","Type":"ContainerStarted","Data":"a569bcb56a8c118cc1ac31a0ec8ee39435e0e7f8cd0bc3eb866545889864af32"}
Oct 30 23:17:30.525669 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:30.525619    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd" podStartSLOduration=1.5256042459999999 podStartE2EDuration="1.525604246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:17:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:17:30.523650928 +0000 UTC m=+4235.880017224" watchObservedRunningTime="2026-10-30 23:17:30.525604246 +0000 UTC m=+4235.881970561"
Oct 30 23:17:30.528150 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:30.528129    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"
Oct 30 23:17:35.129992 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:35.129946    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:17:35.300418 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:35.300282    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:17:39.528323 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:39.528277    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"
Oct 30 23:17:39.541057 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:39.541028    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"
Oct 30 23:17:39.584715 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:39.584679    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"
Oct 30 23:17:40.302524 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:17:40.302488    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:17:53.356430 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:53.356392    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:17:57.003116 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:57.003082    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"]
Oct 30 23:17:57.117377 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:17:57.117339    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:18:01.344251 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:01.316449    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"]
Oct 30 23:18:01.389024 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:01.388689    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:18:10.300275 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:18:10.300174    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:18:12.024473 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:12.024415    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7" podUID="a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f" containerName="main" containerID="cri-o://72a0b9104307a0b84e78056f57aef5a6b7a26098558e996fcbdcd4763b3ee1ce" gracePeriod=45
Oct 30 23:18:12.279860 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:12.279803    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"
Oct 30 23:18:12.315472 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:12.315442    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-dshm\") pod \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\" (UID: \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\") "
Oct 30 23:18:12.315631 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:12.315488    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-kube-api-access-5722j\" (UniqueName: \"kubernetes.io/projected/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-kube-api-access-5722j\") pod \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\" (UID: \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\") "
Oct 30 23:18:12.315631 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:12.315566    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-tmp-dir\") pod \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\" (UID: \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\") "
Oct 30 23:18:12.315631 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:12.315606    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-home\" (UniqueName: \"kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-home\") pod \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\" (UID: \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\") "
Oct 30 23:18:12.315820 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:12.315695    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-tls-certs\" (UniqueName: \"kubernetes.io/secret/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-tls-certs\") pod \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\" (UID: \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\") "
Oct 30 23:18:12.315820 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:12.315756    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-model-cache\") pod \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\" (UID: \"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f\") "
Oct 30 23:18:12.315932 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:12.315887    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-tmp-dir" pod "a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f" (UID: "a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:18:12.316002 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:12.315970    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-home" pod "a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f" (UID: "a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:18:12.316062 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:12.315997    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-model-cache" pod "a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f" (UID: "a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:18:12.316119 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:12.316105    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:18:12.316177 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:12.316126    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:18:12.316177 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:12.316139    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:18:12.318045 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:12.318026    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-kube-api-access-5722j" pod "a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f" (UID: "a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f"). InnerVolumeSpecName "kube-api-access-5722j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:18:12.318129 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:12.318040    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-tls-certs" pod "a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f" (UID: "a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:18:12.318176 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:12.318137    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-dshm" pod "a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f" (UID: "a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:18:12.416890 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:12.416856    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:18:12.416890 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:12.416884    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:18:12.416890 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:12.416894    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5722j\" (UniqueName: \"kubernetes.io/projected/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f-kube-api-access-5722j\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:18:12.946944 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:12.946907    4889 generic.go:356] "Generic (PLEG): container finished" podID="a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f" containerID="72a0b9104307a0b84e78056f57aef5a6b7a26098558e996fcbdcd4763b3ee1ce" exitCode=0
Oct 30 23:18:12.946944 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:12.946945    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"
Oct 30 23:18:12.947186 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:12.946953    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7" event={"ID":"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f","Type":"ContainerDied","Data":"72a0b9104307a0b84e78056f57aef5a6b7a26098558e996fcbdcd4763b3ee1ce"}
Oct 30 23:18:12.947186 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:12.946989    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7" event={"ID":"a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f","Type":"ContainerDied","Data":"698ae50120f2a7c4308e380393b49f97fd5ff6798205515efccb384a3293618a"}
Oct 30 23:18:12.947186 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:12.947010    4889 scope.go:122] "RemoveContainer" containerID="72a0b9104307a0b84e78056f57aef5a6b7a26098558e996fcbdcd4763b3ee1ce"
Oct 30 23:18:12.955911 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:12.955883    4889 scope.go:122] "RemoveContainer" containerID="72a0b9104307a0b84e78056f57aef5a6b7a26098558e996fcbdcd4763b3ee1ce"
Oct 30 23:18:12.956192 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:18:12.956169    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72a0b9104307a0b84e78056f57aef5a6b7a26098558e996fcbdcd4763b3ee1ce\": container with ID starting with 72a0b9104307a0b84e78056f57aef5a6b7a26098558e996fcbdcd4763b3ee1ce not found: ID does not exist" containerID="72a0b9104307a0b84e78056f57aef5a6b7a26098558e996fcbdcd4763b3ee1ce"
Oct 30 23:18:12.956299 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:12.956200    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72a0b9104307a0b84e78056f57aef5a6b7a26098558e996fcbdcd4763b3ee1ce"} err="failed to get container status \"72a0b9104307a0b84e78056f57aef5a6b7a26098558e996fcbdcd4763b3ee1ce\": rpc error: code = NotFound desc = could not find container \"72a0b9104307a0b84e78056f57aef5a6b7a26098558e996fcbdcd4763b3ee1ce\": container with ID starting with 72a0b9104307a0b84e78056f57aef5a6b7a26098558e996fcbdcd4763b3ee1ce not found: ID does not exist"
Oct 30 23:18:13.001578 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:13.001551    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"]
Oct 30 23:18:13.015135 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:13.015100    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-2917e5f5-kserve-74c68b675c-dnhh7"]
Oct 30 23:18:14.772554 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:14.772523    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f" path="/var/lib/kubelet/pods/a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f/volumes"
Oct 30 23:18:16.642910 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:16.642862    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd" podUID="75d65a47-3e5b-4099-890c-f7b5cbdb5654" containerName="main" containerID="cri-o://b1e1c44b667909f515d705a0e23bdedb60096b5dd514b1a1615fdb84a51ea7bd" gracePeriod=45
Oct 30 23:18:17.087946 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:17.087668    4889 generic.go:356] "Generic (PLEG): container finished" podID="75d65a47-3e5b-4099-890c-f7b5cbdb5654" containerID="b1e1c44b667909f515d705a0e23bdedb60096b5dd514b1a1615fdb84a51ea7bd" exitCode=0
Oct 30 23:18:17.087946 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:17.087830    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd" event={"ID":"75d65a47-3e5b-4099-890c-f7b5cbdb5654","Type":"ContainerDied","Data":"b1e1c44b667909f515d705a0e23bdedb60096b5dd514b1a1615fdb84a51ea7bd"}
Oct 30 23:18:17.780892 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:17.780853    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"
Oct 30 23:18:18.032251 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:18.028138    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75d65a47-3e5b-4099-890c-f7b5cbdb5654-tls-certs\" (UniqueName: \"kubernetes.io/secret/75d65a47-3e5b-4099-890c-f7b5cbdb5654-tls-certs\") pod \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\" (UID: \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\") "
Oct 30 23:18:18.032251 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:18.028192    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75d65a47-3e5b-4099-890c-f7b5cbdb5654-kube-api-access-7klff\" (UniqueName: \"kubernetes.io/projected/75d65a47-3e5b-4099-890c-f7b5cbdb5654-kube-api-access-7klff\") pod \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\" (UID: \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\") "
Oct 30 23:18:18.032251 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:18.028305    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-dshm\" (UniqueName: \"kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-dshm\") pod \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\" (UID: \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\") "
Oct 30 23:18:18.032251 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:18.028359    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-tmp-dir\") pod \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\" (UID: \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\") "
Oct 30 23:18:18.032251 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:18.028409    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-model-cache\") pod \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\" (UID: \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\") "
Oct 30 23:18:18.032251 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:18.028442    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-home\" (UniqueName: \"kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-home\") pod \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\" (UID: \"75d65a47-3e5b-4099-890c-f7b5cbdb5654\") "
Oct 30 23:18:18.036253 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:18.032922    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-home" pod "75d65a47-3e5b-4099-890c-f7b5cbdb5654" (UID: "75d65a47-3e5b-4099-890c-f7b5cbdb5654"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:18:18.044486 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:18.044455    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-tmp-dir" pod "75d65a47-3e5b-4099-890c-f7b5cbdb5654" (UID: "75d65a47-3e5b-4099-890c-f7b5cbdb5654"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:18:18.045046 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:18.045023    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-model-cache" pod "75d65a47-3e5b-4099-890c-f7b5cbdb5654" (UID: "75d65a47-3e5b-4099-890c-f7b5cbdb5654"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:18:18.070051 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:18.070015    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d65a47-3e5b-4099-890c-f7b5cbdb5654-kube-api-access-7klff" pod "75d65a47-3e5b-4099-890c-f7b5cbdb5654" (UID: "75d65a47-3e5b-4099-890c-f7b5cbdb5654"). InnerVolumeSpecName "kube-api-access-7klff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:18:18.103906 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:18.103751    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75d65a47-3e5b-4099-890c-f7b5cbdb5654-tls-certs" pod "75d65a47-3e5b-4099-890c-f7b5cbdb5654" (UID: "75d65a47-3e5b-4099-890c-f7b5cbdb5654"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:18:18.103906 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:18.103758    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-dshm" pod "75d65a47-3e5b-4099-890c-f7b5cbdb5654" (UID: "75d65a47-3e5b-4099-890c-f7b5cbdb5654"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:18:18.122258 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:18.110401    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd" event={"ID":"75d65a47-3e5b-4099-890c-f7b5cbdb5654","Type":"ContainerDied","Data":"a569bcb56a8c118cc1ac31a0ec8ee39435e0e7f8cd0bc3eb866545889864af32"}
Oct 30 23:18:18.122258 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:18.110461    4889 scope.go:122] "RemoveContainer" containerID="b1e1c44b667909f515d705a0e23bdedb60096b5dd514b1a1615fdb84a51ea7bd"
Oct 30 23:18:18.122258 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:18.110608    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"
Oct 30 23:18:18.161249 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:18.142701    4889 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-dshm\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:18:18.161839 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:18.161816    4889 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-tmp-dir\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:18:18.161991 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:18.161978    4889 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-model-cache\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:18:18.162095 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:18.162084    4889 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75d65a47-3e5b-4099-890c-f7b5cbdb5654-home\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:18:18.162191 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:18.162180    4889 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75d65a47-3e5b-4099-890c-f7b5cbdb5654-tls-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:18:18.162324 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:18.162310    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7klff\" (UniqueName: \"kubernetes.io/projected/75d65a47-3e5b-4099-890c-f7b5cbdb5654-kube-api-access-7klff\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:18:18.250549 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:18.250511    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"]
Oct 30 23:18:18.269250 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:18.253639    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd"]
Oct 30 23:18:18.582407 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:18.582360    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-2917e5f5-kserve-5bf5f67f7c-jzldd" podUID="75d65a47-3e5b-4099-890c-f7b5cbdb5654" containerName="main" probeResult="failure" output="Get \"https://10.132.1.24:8000/ready\": context deadline exceeded"
Oct 30 23:18:18.773675 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:18.773646    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75d65a47-3e5b-4099-890c-f7b5cbdb5654" path="/var/lib/kubelet/pods/75d65a47-3e5b-4099-890c-f7b5cbdb5654/volumes"
Oct 30 23:18:40.304520 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:18:40.304477    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:18:41.102150 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:41.102107    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-567764f58f-qdbbq"]
Oct 30 23:18:41.102970 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:41.102938    4889 cpu_manager.go:414] "containerMap: removing container" podUID="75d65a47-3e5b-4099-890c-f7b5cbdb5654" containerName="main"
Oct 30 23:18:41.102970 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:41.102971    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75d65a47-3e5b-4099-890c-f7b5cbdb5654" containerName="main"
Oct 30 23:18:41.103239 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:41.103012    4889 cpu_manager.go:414] "containerMap: removing container" podUID="a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f" containerName="main"
Oct 30 23:18:41.103239 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:41.103021    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f" containerName="main"
Oct 30 23:18:41.103239 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:41.103172    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9fc8cad-c0ac-4b7d-a716-9bb3319f7f8f" containerName="main"
Oct 30 23:18:41.103239 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:41.103207    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="75d65a47-3e5b-4099-890c-f7b5cbdb5654" containerName="main"
Oct 30 23:18:41.107300 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:41.107278    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-567764f58f-qdbbq"
Oct 30 23:18:41.123002 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:41.122975    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-567764f58f-qdbbq"]
Oct 30 23:18:41.247149 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:41.247117    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6e1d0c3-54bf-490d-bb4e-9bd7039098cf-metrics-certs\") pod \"maas-api-567764f58f-qdbbq\" (UID: \"e6e1d0c3-54bf-490d-bb4e-9bd7039098cf\") " pod="odh-ai-gateway-infra/maas-api-567764f58f-qdbbq"
Oct 30 23:18:41.247360 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:41.247268    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtwrc\" (UniqueName: \"kubernetes.io/projected/e6e1d0c3-54bf-490d-bb4e-9bd7039098cf-kube-api-access-jtwrc\") pod \"maas-api-567764f58f-qdbbq\" (UID: \"e6e1d0c3-54bf-490d-bb4e-9bd7039098cf\") " pod="odh-ai-gateway-infra/maas-api-567764f58f-qdbbq"
Oct 30 23:18:41.247360 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:41.247306    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6e1d0c3-54bf-490d-bb4e-9bd7039098cf-openshift-service-ca\") pod \"maas-api-567764f58f-qdbbq\" (UID: \"e6e1d0c3-54bf-490d-bb4e-9bd7039098cf\") " pod="odh-ai-gateway-infra/maas-api-567764f58f-qdbbq"
Oct 30 23:18:41.247452 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:41.247406    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6e1d0c3-54bf-490d-bb4e-9bd7039098cf-maas-api-tls\") pod \"maas-api-567764f58f-qdbbq\" (UID: \"e6e1d0c3-54bf-490d-bb4e-9bd7039098cf\") " pod="odh-ai-gateway-infra/maas-api-567764f58f-qdbbq"
Oct 30 23:18:41.349012 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:41.348976    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtwrc\" (UniqueName: \"kubernetes.io/projected/e6e1d0c3-54bf-490d-bb4e-9bd7039098cf-kube-api-access-jtwrc\") pod \"maas-api-567764f58f-qdbbq\" (UID: \"e6e1d0c3-54bf-490d-bb4e-9bd7039098cf\") " pod="odh-ai-gateway-infra/maas-api-567764f58f-qdbbq"
Oct 30 23:18:41.349012 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:41.349013    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6e1d0c3-54bf-490d-bb4e-9bd7039098cf-openshift-service-ca\") pod \"maas-api-567764f58f-qdbbq\" (UID: \"e6e1d0c3-54bf-490d-bb4e-9bd7039098cf\") " pod="odh-ai-gateway-infra/maas-api-567764f58f-qdbbq"
Oct 30 23:18:41.349506 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:41.349077    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6e1d0c3-54bf-490d-bb4e-9bd7039098cf-maas-api-tls\") pod \"maas-api-567764f58f-qdbbq\" (UID: \"e6e1d0c3-54bf-490d-bb4e-9bd7039098cf\") " pod="odh-ai-gateway-infra/maas-api-567764f58f-qdbbq"
Oct 30 23:18:41.349506 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:41.349104    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6e1d0c3-54bf-490d-bb4e-9bd7039098cf-metrics-certs\") pod \"maas-api-567764f58f-qdbbq\" (UID: \"e6e1d0c3-54bf-490d-bb4e-9bd7039098cf\") " pod="odh-ai-gateway-infra/maas-api-567764f58f-qdbbq"
Oct 30 23:18:41.349736 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:41.349716    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6e1d0c3-54bf-490d-bb4e-9bd7039098cf-openshift-service-ca\") pod \"maas-api-567764f58f-qdbbq\" (UID: \"e6e1d0c3-54bf-490d-bb4e-9bd7039098cf\") " pod="odh-ai-gateway-infra/maas-api-567764f58f-qdbbq"
Oct 30 23:18:41.351852 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:41.351825    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6e1d0c3-54bf-490d-bb4e-9bd7039098cf-metrics-certs\") pod \"maas-api-567764f58f-qdbbq\" (UID: \"e6e1d0c3-54bf-490d-bb4e-9bd7039098cf\") " pod="odh-ai-gateway-infra/maas-api-567764f58f-qdbbq"
Oct 30 23:18:41.352063 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:41.352043    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6e1d0c3-54bf-490d-bb4e-9bd7039098cf-maas-api-tls\") pod \"maas-api-567764f58f-qdbbq\" (UID: \"e6e1d0c3-54bf-490d-bb4e-9bd7039098cf\") " pod="odh-ai-gateway-infra/maas-api-567764f58f-qdbbq"
Oct 30 23:18:41.370091 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:41.370029    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtwrc\" (UniqueName: \"kubernetes.io/projected/e6e1d0c3-54bf-490d-bb4e-9bd7039098cf-kube-api-access-jtwrc\") pod \"maas-api-567764f58f-qdbbq\" (UID: \"e6e1d0c3-54bf-490d-bb4e-9bd7039098cf\") " pod="odh-ai-gateway-infra/maas-api-567764f58f-qdbbq"
Oct 30 23:18:41.425194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:41.425170    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-567764f58f-qdbbq"
Oct 30 23:18:41.721141 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:41.721116    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-567764f58f-qdbbq"]
Oct 30 23:18:41.722447 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:18:41.722418    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e1d0c3_54bf_490d_bb4e_9bd7039098cf.slice/crio-ce2ac43c65e80b7b209068c7bce9e7506d798ef8a0bf7d9a9171557f17766ceb WatchSource:0}: Error finding container ce2ac43c65e80b7b209068c7bce9e7506d798ef8a0bf7d9a9171557f17766ceb: Status 404 returned error can't find the container with id ce2ac43c65e80b7b209068c7bce9e7506d798ef8a0bf7d9a9171557f17766ceb
Oct 30 23:18:41.724133 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:41.724113    4889 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:18:42.247081 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:42.247044    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-567764f58f-qdbbq" event={"ID":"e6e1d0c3-54bf-490d-bb4e-9bd7039098cf","Type":"ContainerStarted","Data":"7a120f020211733476e9e9d819daff802ba5f58ca86df7ccb62949766fcfc34f"}
Oct 30 23:18:42.247287 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:42.247106    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-567764f58f-qdbbq" event={"ID":"e6e1d0c3-54bf-490d-bb4e-9bd7039098cf","Type":"ContainerStarted","Data":"ce2ac43c65e80b7b209068c7bce9e7506d798ef8a0bf7d9a9171557f17766ceb"}
Oct 30 23:18:42.247287 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:42.247195    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-567764f58f-qdbbq"
Oct 30 23:18:42.290140 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:42.290091    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-567764f58f-qdbbq" podStartSLOduration=0.878747846 podStartE2EDuration="1.290076144s" totalImagesPullingTime="411.328298ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:18:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:42.288646634 +0000 UTC m=+4307.645012947" watchObservedRunningTime="2026-10-30 23:18:42.290076144 +0000 UTC m=+4307.646442458"
Oct 30 23:18:48.161336 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.161296    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:18:48.257361 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.257329    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-567764f58f-qdbbq"
Oct 30 23:18:48.542436 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.542391    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"]
Oct 30 23:18:48.547171 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.547148    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.581279 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.581217    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1595eccb-openshift-default-dockercfg-2mtrf\""
Oct 30 23:18:48.583396 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.583368    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"]
Oct 30 23:18:48.704000 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.703959    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:18:48.723161 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.723125    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrpl2\" (UniqueName: \"kubernetes.io/projected/223b2134-2c18-4e58-aa61-c747755ea925-kube-api-access-qrpl2\") pod \"e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") " pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.723337 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.723237    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/223b2134-2c18-4e58-aa61-c747755ea925-istiod-ca-cert\") pod \"e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") " pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.723337 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.723275    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-istio-data\") pod \"e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") " pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.723337 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.723308    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-credential-socket\") pod \"e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") " pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.723469 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.723350    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-workload-socket\") pod \"e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") " pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.723469 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.723378    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/223b2134-2c18-4e58-aa61-c747755ea925-istio-token\") pod \"e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") " pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.723546 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.723473    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-istio-envoy\") pod \"e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") " pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.723546 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.723501    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-workload-certs\") pod \"e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") " pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.723546 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.723535    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/223b2134-2c18-4e58-aa61-c747755ea925-istio-podinfo\") pod \"e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") " pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.824522 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.824422    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-istio-data\") pod \"e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") " pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.824522 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.824473    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-credential-socket\") pod \"e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") " pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.824522 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.824512    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-workload-socket\") pod \"e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") " pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.824854 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.824538    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/223b2134-2c18-4e58-aa61-c747755ea925-istio-token\") pod \"e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") " pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.824854 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.824605    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-istio-envoy\") pod \"e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") " pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.824854 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.824642    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-workload-certs\") pod \"e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") " pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.824854 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.824689    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/223b2134-2c18-4e58-aa61-c747755ea925-istio-podinfo\") pod \"e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") " pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.824854 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.824724    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrpl2\" (UniqueName: \"kubernetes.io/projected/223b2134-2c18-4e58-aa61-c747755ea925-kube-api-access-qrpl2\") pod \"e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") " pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.824854 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.824799    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/223b2134-2c18-4e58-aa61-c747755ea925-istiod-ca-cert\") pod \"e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") " pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.825238 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.824866    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-istio-data\") pod \"e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") " pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.825238 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.824932    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-credential-socket\") pod \"e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") " pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.825238 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.824991    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-workload-certs\") pod \"e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") " pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.825238 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.825180    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-workload-socket\") pod \"e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") " pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.825630 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.825608    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/223b2134-2c18-4e58-aa61-c747755ea925-istiod-ca-cert\") pod \"e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") " pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.827101 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.827082    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-istio-envoy\") pod \"e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") " pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.827199 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.827169    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/223b2134-2c18-4e58-aa61-c747755ea925-istio-podinfo\") pod \"e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") " pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.846103 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.846069    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/223b2134-2c18-4e58-aa61-c747755ea925-istio-token\") pod \"e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") " pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.846249 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.846192    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrpl2\" (UniqueName: \"kubernetes.io/projected/223b2134-2c18-4e58-aa61-c747755ea925-kube-api-access-qrpl2\") pod \"e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") " pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.862580 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.862562    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:48.890688 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:48.890656    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:18:49.179833 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:49.179809    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"]
Oct 30 23:18:49.181059 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:18:49.181031    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223b2134_2c18_4e58_aa61_c747755ea925.slice/crio-38e8830c4eb64dd5e879bfd0ef42e098cadb720097c8edcd418bc1f507c177bc WatchSource:0}: Error finding container 38e8830c4eb64dd5e879bfd0ef42e098cadb720097c8edcd418bc1f507c177bc: Status 404 returned error can't find the container with id 38e8830c4eb64dd5e879bfd0ef42e098cadb720097c8edcd418bc1f507c177bc
Oct 30 23:18:49.183209 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:49.183169    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:18:49.183312 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:49.183253    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:18:49.183312 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:49.183285    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:18:49.282019 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:49.281981    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb" event={"ID":"223b2134-2c18-4e58-aa61-c747755ea925","Type":"ContainerStarted","Data":"724d5f3faad685f6c9bc8236a6ff04e6ff69caf01aa30ab23c051afef24f6587"}
Oct 30 23:18:49.282167 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:49.282050    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb" event={"ID":"223b2134-2c18-4e58-aa61-c747755ea925","Type":"ContainerStarted","Data":"38e8830c4eb64dd5e879bfd0ef42e098cadb720097c8edcd418bc1f507c177bc"}
Oct 30 23:18:49.328840 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:49.328780    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb" podStartSLOduration=1.328765268 podStartE2EDuration="1.328765268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:49.328504848 +0000 UTC m=+4314.684871176" watchObservedRunningTime="2026-10-30 23:18:49.328765268 +0000 UTC m=+4314.685131577"
Oct 30 23:18:49.862961 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:49.862931    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:50.868324 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:50.868295    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:51.006419 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:51.006372    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:18:51.192544 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:51.192444    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:18:51.291448 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:51.291422    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:51.292513 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:51.292493    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:18:53.271320 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.271285    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:18:53.321511 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.321476    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx"]
Oct 30 23:18:53.325185 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.325164    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx"
Oct 30 23:18:53.343673 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.343647    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx"]
Oct 30 23:18:53.355200 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.355176    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1595eccb\""
Oct 30 23:18:53.355366 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.355263    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1595eccb-dockercfg-6rrln\""
Oct 30 23:18:53.366082 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.365890    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg48j\" (UniqueName: \"kubernetes.io/projected/3481f93a-3c14-43b5-a79c-33e60eaafb03-kube-api-access-dg48j\") pod \"payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx\" (UID: \"3481f93a-3c14-43b5-a79c-33e60eaafb03\") " pod="openshift-ingress/payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx"
Oct 30 23:18:53.366669 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.366360    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3481f93a-3c14-43b5-a79c-33e60eaafb03-plugins-config-volume\") pod \"payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx\" (UID: \"3481f93a-3c14-43b5-a79c-33e60eaafb03\") " pod="openshift-ingress/payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx"
Oct 30 23:18:53.372078 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.372056    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7"]
Oct 30 23:18:53.376466 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.376450    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7"
Oct 30 23:18:53.391365 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.391341    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7"]
Oct 30 23:18:53.455724 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.455565    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:18:53.467212 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.467171    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg48j\" (UniqueName: \"kubernetes.io/projected/3481f93a-3c14-43b5-a79c-33e60eaafb03-kube-api-access-dg48j\") pod \"payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx\" (UID: \"3481f93a-3c14-43b5-a79c-33e60eaafb03\") " pod="openshift-ingress/payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx"
Oct 30 23:18:53.467453 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.467302    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3481f93a-3c14-43b5-a79c-33e60eaafb03-plugins-config-volume\") pod \"payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx\" (UID: \"3481f93a-3c14-43b5-a79c-33e60eaafb03\") " pod="openshift-ingress/payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx"
Oct 30 23:18:53.467453 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.467345    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtpfw\" (UniqueName: \"kubernetes.io/projected/1908b3ff-74fa-4341-93f3-1db56d7a9271-kube-api-access-gtpfw\") pod \"payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7\" (UID: \"1908b3ff-74fa-4341-93f3-1db56d7a9271\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7"
Oct 30 23:18:53.467453 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.467397    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1908b3ff-74fa-4341-93f3-1db56d7a9271-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7\" (UID: \"1908b3ff-74fa-4341-93f3-1db56d7a9271\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7"
Oct 30 23:18:53.468555 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.468528    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3481f93a-3c14-43b5-a79c-33e60eaafb03-plugins-config-volume\") pod \"payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx\" (UID: \"3481f93a-3c14-43b5-a79c-33e60eaafb03\") " pod="openshift-ingress/payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx"
Oct 30 23:18:53.488212 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.488186    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg48j\" (UniqueName: \"kubernetes.io/projected/3481f93a-3c14-43b5-a79c-33e60eaafb03-kube-api-access-dg48j\") pod \"payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx\" (UID: \"3481f93a-3c14-43b5-a79c-33e60eaafb03\") " pod="openshift-ingress/payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx"
Oct 30 23:18:53.568690 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.568581    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtpfw\" (UniqueName: \"kubernetes.io/projected/1908b3ff-74fa-4341-93f3-1db56d7a9271-kube-api-access-gtpfw\") pod \"payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7\" (UID: \"1908b3ff-74fa-4341-93f3-1db56d7a9271\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7"
Oct 30 23:18:53.568690 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.568618    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1908b3ff-74fa-4341-93f3-1db56d7a9271-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7\" (UID: \"1908b3ff-74fa-4341-93f3-1db56d7a9271\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7"
Oct 30 23:18:53.569495 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.569469    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1908b3ff-74fa-4341-93f3-1db56d7a9271-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7\" (UID: \"1908b3ff-74fa-4341-93f3-1db56d7a9271\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7"
Oct 30 23:18:53.590768 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.590738    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtpfw\" (UniqueName: \"kubernetes.io/projected/1908b3ff-74fa-4341-93f3-1db56d7a9271-kube-api-access-gtpfw\") pod \"payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7\" (UID: \"1908b3ff-74fa-4341-93f3-1db56d7a9271\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7"
Oct 30 23:18:53.624422 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.624392    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:18:53.639308 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.639284    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx"
Oct 30 23:18:53.691611 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.691579    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7"
Oct 30 23:18:53.929073 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.928997    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn"]
Oct 30 23:18:53.933809 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.933773    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn"
Oct 30 23:18:53.935196 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.935177    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx"]
Oct 30 23:18:53.936004 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:18:53.935981    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3481f93a_3c14_43b5_a79c_33e60eaafb03.slice/crio-efde2a7cc3371a01a0794bb93c9494e5afa3f17c5022060352939c15c977bca9 WatchSource:0}: Error finding container efde2a7cc3371a01a0794bb93c9494e5afa3f17c5022060352939c15c977bca9: Status 404 returned error can't find the container with id efde2a7cc3371a01a0794bb93c9494e5afa3f17c5022060352939c15c977bca9
Oct 30 23:18:53.954164 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.954134    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1595eccb-serving-cert\""
Oct 30 23:18:53.957906 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.957217    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn"]
Oct 30 23:18:53.972732 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.972683    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6df8663-a978-4719-9322-51007ab6b5af-openshift-service-ca\") pod \"maas-api-e2e-ait-1595eccb-846d486656-fv4fn\" (UID: \"d6df8663-a978-4719-9322-51007ab6b5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn"
Oct 30 23:18:53.972885 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.972753    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6df8663-a978-4719-9322-51007ab6b5af-metrics-certs\") pod \"maas-api-e2e-ait-1595eccb-846d486656-fv4fn\" (UID: \"d6df8663-a978-4719-9322-51007ab6b5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn"
Oct 30 23:18:53.972885 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.972860    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6df8663-a978-4719-9322-51007ab6b5af-maas-api-tls\") pod \"maas-api-e2e-ait-1595eccb-846d486656-fv4fn\" (UID: \"d6df8663-a978-4719-9322-51007ab6b5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn"
Oct 30 23:18:53.973013 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:53.972995    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgh9z\" (UniqueName: \"kubernetes.io/projected/d6df8663-a978-4719-9322-51007ab6b5af-kube-api-access-xgh9z\") pod \"maas-api-e2e-ait-1595eccb-846d486656-fv4fn\" (UID: \"d6df8663-a978-4719-9322-51007ab6b5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn"
Oct 30 23:18:54.008478 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:54.008456    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7"]
Oct 30 23:18:54.012044 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:18:54.012012    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1908b3ff_74fa_4341_93f3_1db56d7a9271.slice/crio-942e13e391a93010d260f71b8187eb24c1d6790ca4842832f7897de78fedf0ce WatchSource:0}: Error finding container 942e13e391a93010d260f71b8187eb24c1d6790ca4842832f7897de78fedf0ce: Status 404 returned error can't find the container with id 942e13e391a93010d260f71b8187eb24c1d6790ca4842832f7897de78fedf0ce
Oct 30 23:18:54.074195 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:54.074171    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6df8663-a978-4719-9322-51007ab6b5af-maas-api-tls\") pod \"maas-api-e2e-ait-1595eccb-846d486656-fv4fn\" (UID: \"d6df8663-a978-4719-9322-51007ab6b5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn"
Oct 30 23:18:54.074331 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:54.074282    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgh9z\" (UniqueName: \"kubernetes.io/projected/d6df8663-a978-4719-9322-51007ab6b5af-kube-api-access-xgh9z\") pod \"maas-api-e2e-ait-1595eccb-846d486656-fv4fn\" (UID: \"d6df8663-a978-4719-9322-51007ab6b5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn"
Oct 30 23:18:54.074331 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:54.074327    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6df8663-a978-4719-9322-51007ab6b5af-openshift-service-ca\") pod \"maas-api-e2e-ait-1595eccb-846d486656-fv4fn\" (UID: \"d6df8663-a978-4719-9322-51007ab6b5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn"
Oct 30 23:18:54.074448 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:54.074370    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6df8663-a978-4719-9322-51007ab6b5af-metrics-certs\") pod \"maas-api-e2e-ait-1595eccb-846d486656-fv4fn\" (UID: \"d6df8663-a978-4719-9322-51007ab6b5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn"
Oct 30 23:18:54.075138 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:54.075111    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6df8663-a978-4719-9322-51007ab6b5af-openshift-service-ca\") pod \"maas-api-e2e-ait-1595eccb-846d486656-fv4fn\" (UID: \"d6df8663-a978-4719-9322-51007ab6b5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn"
Oct 30 23:18:54.076855 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:54.076827    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6df8663-a978-4719-9322-51007ab6b5af-maas-api-tls\") pod \"maas-api-e2e-ait-1595eccb-846d486656-fv4fn\" (UID: \"d6df8663-a978-4719-9322-51007ab6b5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn"
Oct 30 23:18:54.077003 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:54.076983    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6df8663-a978-4719-9322-51007ab6b5af-metrics-certs\") pod \"maas-api-e2e-ait-1595eccb-846d486656-fv4fn\" (UID: \"d6df8663-a978-4719-9322-51007ab6b5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn"
Oct 30 23:18:54.084837 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:54.084811    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:18:54.094480 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:54.094460    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgh9z\" (UniqueName: \"kubernetes.io/projected/d6df8663-a978-4719-9322-51007ab6b5af-kube-api-access-xgh9z\") pod \"maas-api-e2e-ait-1595eccb-846d486656-fv4fn\" (UID: \"d6df8663-a978-4719-9322-51007ab6b5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn"
Oct 30 23:18:54.250097 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:54.250066    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn"
Oct 30 23:18:54.261641 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:54.261599    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:18:54.317917 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:54.317880    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7" event={"ID":"1908b3ff-74fa-4341-93f3-1db56d7a9271","Type":"ContainerStarted","Data":"ffd607eb635643e0cbb4f8d75dc39499eb0c05bdf0fb205ce8b0bf5c32107ec6"}
Oct 30 23:18:54.318348 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:54.317933    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7" event={"ID":"1908b3ff-74fa-4341-93f3-1db56d7a9271","Type":"ContainerStarted","Data":"942e13e391a93010d260f71b8187eb24c1d6790ca4842832f7897de78fedf0ce"}
Oct 30 23:18:54.318645 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:54.318626    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7"
Oct 30 23:18:54.320365 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:54.320327    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx" event={"ID":"3481f93a-3c14-43b5-a79c-33e60eaafb03","Type":"ContainerStarted","Data":"d907650ff56224d18bbac19eaa8399fc849ede2a62af935fe3fd45e2d79496f7"}
Oct 30 23:18:54.320469 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:54.320380    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx" event={"ID":"3481f93a-3c14-43b5-a79c-33e60eaafb03","Type":"ContainerStarted","Data":"efde2a7cc3371a01a0794bb93c9494e5afa3f17c5022060352939c15c977bca9"}
Oct 30 23:18:54.320833 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:54.320817    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx"
Oct 30 23:18:54.361792 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:54.361731    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7" podStartSLOduration=1.361717991 podStartE2EDuration="1.361717991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:54.359931667 +0000 UTC m=+4319.716297981" watchObservedRunningTime="2026-10-30 23:18:54.361717991 +0000 UTC m=+4319.718084306"
Oct 30 23:18:54.401344 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:54.401297    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx" podStartSLOduration=1.401285638 podStartE2EDuration="1.401285638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:54.400965843 +0000 UTC m=+4319.757332157" watchObservedRunningTime="2026-10-30 23:18:54.401285638 +0000 UTC m=+4319.757651953"
Oct 30 23:18:54.558769 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:54.558733    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn"]
Oct 30 23:18:54.559637 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:18:54.559605    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6df8663_a978_4719_9322_51007ab6b5af.slice/crio-e321222b8b3d28ed50acab01c964ce011a6c34ba71e3e501cc1de188eca68398 WatchSource:0}: Error finding container e321222b8b3d28ed50acab01c964ce011a6c34ba71e3e501cc1de188eca68398: Status 404 returned error can't find the container with id e321222b8b3d28ed50acab01c964ce011a6c34ba71e3e501cc1de188eca68398
Oct 30 23:18:55.326813 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:55.326779    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn" event={"ID":"d6df8663-a978-4719-9322-51007ab6b5af","Type":"ContainerStarted","Data":"893f8e098bb91cc86bffeb6385083b32dccf1ab6358764382bde05d772661a4b"}
Oct 30 23:18:55.327310 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:55.326837    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn" event={"ID":"d6df8663-a978-4719-9322-51007ab6b5af","Type":"ContainerStarted","Data":"e321222b8b3d28ed50acab01c964ce011a6c34ba71e3e501cc1de188eca68398"}
Oct 30 23:18:55.327310 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:55.326866    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn"
Oct 30 23:18:55.370320 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:55.370271    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn" podStartSLOduration=2.014316372 podStartE2EDuration="2.370258631s" totalImagesPullingTime="355.942259ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:18:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:55.369947601 +0000 UTC m=+4320.726313914" watchObservedRunningTime="2026-10-30 23:18:55.370258631 +0000 UTC m=+4320.726624976"
Oct 30 23:18:55.717503 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:55.717400    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:18:55.873868 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:55.873834    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj"]
Oct 30 23:18:55.874071 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:55.874047    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj" podUID="ca14b6fa-378e-42e3-be2e-3b15de4a82cd" containerName="maas-api" containerID="cri-o://a09cc8ffb9c7e4adb1b38db395ad6d8348fe61a45e9404d2ba453844e0b84f80" gracePeriod=30
Oct 30 23:18:57.148839 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:57.148814    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj"
Oct 30 23:18:57.211614 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:57.211524    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-kube-api-access-gz4fg\" (UniqueName: \"kubernetes.io/projected/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-kube-api-access-gz4fg\") pod \"ca14b6fa-378e-42e3-be2e-3b15de4a82cd\" (UID: \"ca14b6fa-378e-42e3-be2e-3b15de4a82cd\") "
Oct 30 23:18:57.211803 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:57.211618    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-maas-api-tls\") pod \"ca14b6fa-378e-42e3-be2e-3b15de4a82cd\" (UID: \"ca14b6fa-378e-42e3-be2e-3b15de4a82cd\") "
Oct 30 23:18:57.211803 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:57.211736    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-metrics-certs\") pod \"ca14b6fa-378e-42e3-be2e-3b15de4a82cd\" (UID: \"ca14b6fa-378e-42e3-be2e-3b15de4a82cd\") "
Oct 30 23:18:57.211936 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:57.211829    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-openshift-service-ca\") pod \"ca14b6fa-378e-42e3-be2e-3b15de4a82cd\" (UID: \"ca14b6fa-378e-42e3-be2e-3b15de4a82cd\") "
Oct 30 23:18:57.212577 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:57.212549    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-openshift-service-ca" pod "ca14b6fa-378e-42e3-be2e-3b15de4a82cd" (UID: "ca14b6fa-378e-42e3-be2e-3b15de4a82cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:18:57.214125 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:57.214096    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-kube-api-access-gz4fg" pod "ca14b6fa-378e-42e3-be2e-3b15de4a82cd" (UID: "ca14b6fa-378e-42e3-be2e-3b15de4a82cd"). InnerVolumeSpecName "kube-api-access-gz4fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:18:57.214683 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:57.214659    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-metrics-certs" pod "ca14b6fa-378e-42e3-be2e-3b15de4a82cd" (UID: "ca14b6fa-378e-42e3-be2e-3b15de4a82cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:18:57.214683 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:57.214671    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-maas-api-tls" pod "ca14b6fa-378e-42e3-be2e-3b15de4a82cd" (UID: "ca14b6fa-378e-42e3-be2e-3b15de4a82cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:18:57.313698 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:57.313657    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:18:57.313698 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:57.313690    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz4fg\" (UniqueName: \"kubernetes.io/projected/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-kube-api-access-gz4fg\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:18:57.313698 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:57.313703    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:18:57.313933 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:57.313715    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca14b6fa-378e-42e3-be2e-3b15de4a82cd-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:18:57.337487 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:57.337451    4889 generic.go:356] "Generic (PLEG): container finished" podID="ca14b6fa-378e-42e3-be2e-3b15de4a82cd" containerID="a09cc8ffb9c7e4adb1b38db395ad6d8348fe61a45e9404d2ba453844e0b84f80" exitCode=0
Oct 30 23:18:57.337613 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:57.337489    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj"
Oct 30 23:18:57.337613 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:57.337526    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj" event={"ID":"ca14b6fa-378e-42e3-be2e-3b15de4a82cd","Type":"ContainerDied","Data":"a09cc8ffb9c7e4adb1b38db395ad6d8348fe61a45e9404d2ba453844e0b84f80"}
Oct 30 23:18:57.337613 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:57.337578    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj" event={"ID":"ca14b6fa-378e-42e3-be2e-3b15de4a82cd","Type":"ContainerDied","Data":"de8a814b2b27382ff3cc9b90b088ba8981e7d3ceb05cc21506824251f4d39c5e"}
Oct 30 23:18:57.337613 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:57.337601    4889 scope.go:122] "RemoveContainer" containerID="a09cc8ffb9c7e4adb1b38db395ad6d8348fe61a45e9404d2ba453844e0b84f80"
Oct 30 23:18:57.348127 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:57.348106    4889 scope.go:122] "RemoveContainer" containerID="a09cc8ffb9c7e4adb1b38db395ad6d8348fe61a45e9404d2ba453844e0b84f80"
Oct 30 23:18:57.348420 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:18:57.348400    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a09cc8ffb9c7e4adb1b38db395ad6d8348fe61a45e9404d2ba453844e0b84f80\": container with ID starting with a09cc8ffb9c7e4adb1b38db395ad6d8348fe61a45e9404d2ba453844e0b84f80 not found: ID does not exist" containerID="a09cc8ffb9c7e4adb1b38db395ad6d8348fe61a45e9404d2ba453844e0b84f80"
Oct 30 23:18:57.348507 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:57.348431    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a09cc8ffb9c7e4adb1b38db395ad6d8348fe61a45e9404d2ba453844e0b84f80"} err="failed to get container status \"a09cc8ffb9c7e4adb1b38db395ad6d8348fe61a45e9404d2ba453844e0b84f80\": rpc error: code = NotFound desc = could not find container \"a09cc8ffb9c7e4adb1b38db395ad6d8348fe61a45e9404d2ba453844e0b84f80\": container with ID starting with a09cc8ffb9c7e4adb1b38db395ad6d8348fe61a45e9404d2ba453844e0b84f80 not found: ID does not exist"
Oct 30 23:18:57.398420 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:57.398390    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj"]
Oct 30 23:18:57.400075 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:57.400051    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cf6b4c745-2jdfj"]
Oct 30 23:18:57.431994 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:57.431961    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:18:58.773281 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:58.773248    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca14b6fa-378e-42e3-be2e-3b15de4a82cd" path="/var/lib/kubelet/pods/ca14b6fa-378e-42e3-be2e-3b15de4a82cd/volumes"
Oct 30 23:18:59.037618 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:18:59.037533    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:00.610841 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:00.610787    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:01.337025 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:01.336998    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn"
Oct 30 23:19:02.244313 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:02.244273    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:04.210416 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:04.210378    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:05.847315 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:05.847274    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:06.337466 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:06.337433    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx"
Oct 30 23:19:06.337748 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:06.337716    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7"
Oct 30 23:19:07.637267 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:07.637191    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:09.392752 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:09.392568    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:10.300379 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:19:10.300329    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:19:11.154445 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:11.154362    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:11.411245 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:11.411144    4889 generic.go:356] "Generic (PLEG): container finished" podID="223b2134-2c18-4e58-aa61-c747755ea925" containerID="724d5f3faad685f6c9bc8236a6ff04e6ff69caf01aa30ab23c051afef24f6587" exitCode=0
Oct 30 23:19:11.411245 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:11.411231    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb" event={"ID":"223b2134-2c18-4e58-aa61-c747755ea925","Type":"ContainerDied","Data":"724d5f3faad685f6c9bc8236a6ff04e6ff69caf01aa30ab23c051afef24f6587"}
Oct 30 23:19:11.411614 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:11.411599    4889 scope.go:122] "RemoveContainer" containerID="724d5f3faad685f6c9bc8236a6ff04e6ff69caf01aa30ab23c051afef24f6587"
Oct 30 23:19:11.412902 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:11.412878    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:19:11.412981 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:11.412949    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:19:11.413030 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:11.413000    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:19:12.443793 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:12.443747    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb" event={"ID":"223b2134-2c18-4e58-aa61-c747755ea925","Type":"ContainerStarted","Data":"9f56737463aa74ce760282ba4ab1574f30be524653490a5d4d433255e6d2271b"}
Oct 30 23:19:12.863295 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:12.863263    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:19:12.863502 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:12.863482    4889 patch_prober.go:29] interesting pod/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 30 23:19:12.863549 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:12.863535    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb" podUID="223b2134-2c18-4e58-aa61-c747755ea925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 30 23:19:12.895897 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:12.895866    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:13.863149 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:13.863115    4889 patch_prober.go:29] interesting pod/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 30 23:19:13.863574 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:13.863167    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb" podUID="223b2134-2c18-4e58-aa61-c747755ea925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 30 23:19:14.751728 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:14.751675    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:14.864058 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:14.864015    4889 patch_prober.go:29] interesting pod/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 30 23:19:14.864510 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:14.864076    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb" podUID="223b2134-2c18-4e58-aa61-c747755ea925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 30 23:19:15.862891 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:15.862856    4889 patch_prober.go:29] interesting pod/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 30 23:19:15.863082 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:15.862906    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb" podUID="223b2134-2c18-4e58-aa61-c747755ea925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 30 23:19:16.484523 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:16.484487    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:16.863973 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:16.863940    4889 patch_prober.go:29] interesting pod/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 30 23:19:16.864158 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:16.864004    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb" podUID="223b2134-2c18-4e58-aa61-c747755ea925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 30 23:19:17.259157 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:17.259123    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx"]
Oct 30 23:19:17.290759 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:17.290731    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7"]
Oct 30 23:19:17.499859 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:17.499816    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:17.692565 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:17.692481    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:17.795173 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:17.795144    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn"]
Oct 30 23:19:17.795415 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:17.795384    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn" podUID="d6df8663-a978-4719-9322-51007ab6b5af" containerName="maas-api" containerID="cri-o://893f8e098bb91cc86bffeb6385083b32dccf1ab6358764382bde05d772661a4b" gracePeriod=30
Oct 30 23:19:17.867545 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:17.867516    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:19:17.867799 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:17.867770    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:19:17.868581 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:17.868560    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:19:17.928715 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:17.928684    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:18.090433 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:18.090411    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn"
Oct 30 23:19:18.268326 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:18.268078    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6df8663-a978-4719-9322-51007ab6b5af-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6df8663-a978-4719-9322-51007ab6b5af-maas-api-tls\") pod \"d6df8663-a978-4719-9322-51007ab6b5af\" (UID: \"d6df8663-a978-4719-9322-51007ab6b5af\") "
Oct 30 23:19:18.268326 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:18.268261    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6df8663-a978-4719-9322-51007ab6b5af-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6df8663-a978-4719-9322-51007ab6b5af-metrics-certs\") pod \"d6df8663-a978-4719-9322-51007ab6b5af\" (UID: \"d6df8663-a978-4719-9322-51007ab6b5af\") "
Oct 30 23:19:18.268580 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:18.268344    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6df8663-a978-4719-9322-51007ab6b5af-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6df8663-a978-4719-9322-51007ab6b5af-openshift-service-ca\") pod \"d6df8663-a978-4719-9322-51007ab6b5af\" (UID: \"d6df8663-a978-4719-9322-51007ab6b5af\") "
Oct 30 23:19:18.268580 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:18.268391    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6df8663-a978-4719-9322-51007ab6b5af-kube-api-access-xgh9z\" (UniqueName: \"kubernetes.io/projected/d6df8663-a978-4719-9322-51007ab6b5af-kube-api-access-xgh9z\") pod \"d6df8663-a978-4719-9322-51007ab6b5af\" (UID: \"d6df8663-a978-4719-9322-51007ab6b5af\") "
Oct 30 23:19:18.270439 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:18.270405    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6df8663-a978-4719-9322-51007ab6b5af-openshift-service-ca" pod "d6df8663-a978-4719-9322-51007ab6b5af" (UID: "d6df8663-a978-4719-9322-51007ab6b5af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:19:18.272632 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:18.272596    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6df8663-a978-4719-9322-51007ab6b5af-maas-api-tls" pod "d6df8663-a978-4719-9322-51007ab6b5af" (UID: "d6df8663-a978-4719-9322-51007ab6b5af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:19:18.277085 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:18.277058    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6df8663-a978-4719-9322-51007ab6b5af-metrics-certs" pod "d6df8663-a978-4719-9322-51007ab6b5af" (UID: "d6df8663-a978-4719-9322-51007ab6b5af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:19:18.278530 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:18.278504    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6df8663-a978-4719-9322-51007ab6b5af-kube-api-access-xgh9z" pod "d6df8663-a978-4719-9322-51007ab6b5af" (UID: "d6df8663-a978-4719-9322-51007ab6b5af"). InnerVolumeSpecName "kube-api-access-xgh9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:19:18.370553 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:18.370514    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6df8663-a978-4719-9322-51007ab6b5af-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:19:18.370553 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:18.370553    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6df8663-a978-4719-9322-51007ab6b5af-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:19:18.370773 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:18.370572    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6df8663-a978-4719-9322-51007ab6b5af-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:19:18.370773 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:18.370586    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgh9z\" (UniqueName: \"kubernetes.io/projected/d6df8663-a978-4719-9322-51007ab6b5af-kube-api-access-xgh9z\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:19:18.476451 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:18.476411    4889 generic.go:356] "Generic (PLEG): container finished" podID="d6df8663-a978-4719-9322-51007ab6b5af" containerID="893f8e098bb91cc86bffeb6385083b32dccf1ab6358764382bde05d772661a4b" exitCode=0
Oct 30 23:19:18.476630 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:18.476456    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn"
Oct 30 23:19:18.476630 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:18.476502    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn" event={"ID":"d6df8663-a978-4719-9322-51007ab6b5af","Type":"ContainerDied","Data":"893f8e098bb91cc86bffeb6385083b32dccf1ab6358764382bde05d772661a4b"}
Oct 30 23:19:18.476630 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:18.476555    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn" event={"ID":"d6df8663-a978-4719-9322-51007ab6b5af","Type":"ContainerDied","Data":"e321222b8b3d28ed50acab01c964ce011a6c34ba71e3e501cc1de188eca68398"}
Oct 30 23:19:18.476630 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:18.476586    4889 scope.go:122] "RemoveContainer" containerID="893f8e098bb91cc86bffeb6385083b32dccf1ab6358764382bde05d772661a4b"
Oct 30 23:19:18.486740 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:18.486710    4889 scope.go:122] "RemoveContainer" containerID="893f8e098bb91cc86bffeb6385083b32dccf1ab6358764382bde05d772661a4b"
Oct 30 23:19:18.487102 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:19:18.487072    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"893f8e098bb91cc86bffeb6385083b32dccf1ab6358764382bde05d772661a4b\": container with ID starting with 893f8e098bb91cc86bffeb6385083b32dccf1ab6358764382bde05d772661a4b not found: ID does not exist" containerID="893f8e098bb91cc86bffeb6385083b32dccf1ab6358764382bde05d772661a4b"
Oct 30 23:19:18.487200 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:18.487107    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"893f8e098bb91cc86bffeb6385083b32dccf1ab6358764382bde05d772661a4b"} err="failed to get container status \"893f8e098bb91cc86bffeb6385083b32dccf1ab6358764382bde05d772661a4b\": rpc error: code = NotFound desc = could not find container \"893f8e098bb91cc86bffeb6385083b32dccf1ab6358764382bde05d772661a4b\": container with ID starting with 893f8e098bb91cc86bffeb6385083b32dccf1ab6358764382bde05d772661a4b not found: ID does not exist"
Oct 30 23:19:18.530423 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:18.530356    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn"]
Oct 30 23:19:18.532546 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:18.532525    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1595eccb-846d486656-fv4fn"]
Oct 30 23:19:18.773005 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:18.772974    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6df8663-a978-4719-9322-51007ab6b5af" path="/var/lib/kubelet/pods/d6df8663-a978-4719-9322-51007ab6b5af/volumes"
Oct 30 23:19:20.106077 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:20.106035    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:20.297843 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:20.297804    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:22.259720 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:22.259671    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx" podUID="3481f93a-3c14-43b5-a79c-33e60eaafb03" containerName="payload-processing" containerID="cri-o://d907650ff56224d18bbac19eaa8399fc849ede2a62af935fe3fd45e2d79496f7" gracePeriod=25
Oct 30 23:19:22.291175 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:22.291145    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7" podUID="1908b3ff-74fa-4341-93f3-1db56d7a9271" containerName="payload-pre-processing" containerID="cri-o://ffd607eb635643e0cbb4f8d75dc39499eb0c05bdf0fb205ce8b0bf5c32107ec6" gracePeriod=25
Oct 30 23:19:22.498202 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:22.498172    4889 generic.go:356] "Generic (PLEG): container finished" podID="1908b3ff-74fa-4341-93f3-1db56d7a9271" containerID="ffd607eb635643e0cbb4f8d75dc39499eb0c05bdf0fb205ce8b0bf5c32107ec6" exitCode=0
Oct 30 23:19:22.498382 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:22.498252    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7" event={"ID":"1908b3ff-74fa-4341-93f3-1db56d7a9271","Type":"ContainerDied","Data":"ffd607eb635643e0cbb4f8d75dc39499eb0c05bdf0fb205ce8b0bf5c32107ec6"}
Oct 30 23:19:22.499968 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:22.499941    4889 generic.go:356] "Generic (PLEG): container finished" podID="3481f93a-3c14-43b5-a79c-33e60eaafb03" containerID="d907650ff56224d18bbac19eaa8399fc849ede2a62af935fe3fd45e2d79496f7" exitCode=0
Oct 30 23:19:22.500105 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:22.500015    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx" event={"ID":"3481f93a-3c14-43b5-a79c-33e60eaafb03","Type":"ContainerDied","Data":"d907650ff56224d18bbac19eaa8399fc849ede2a62af935fe3fd45e2d79496f7"}
Oct 30 23:19:22.534396 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:22.534372    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx"
Oct 30 23:19:22.565295 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:22.565274    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7"
Oct 30 23:19:22.615411 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:22.615377    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1908b3ff-74fa-4341-93f3-1db56d7a9271-kube-api-access-gtpfw\" (UniqueName: \"kubernetes.io/projected/1908b3ff-74fa-4341-93f3-1db56d7a9271-kube-api-access-gtpfw\") pod \"1908b3ff-74fa-4341-93f3-1db56d7a9271\" (UID: \"1908b3ff-74fa-4341-93f3-1db56d7a9271\") "
Oct 30 23:19:22.615583 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:22.615453    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3481f93a-3c14-43b5-a79c-33e60eaafb03-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3481f93a-3c14-43b5-a79c-33e60eaafb03-plugins-config-volume\") pod \"3481f93a-3c14-43b5-a79c-33e60eaafb03\" (UID: \"3481f93a-3c14-43b5-a79c-33e60eaafb03\") "
Oct 30 23:19:22.615583 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:22.615475    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3481f93a-3c14-43b5-a79c-33e60eaafb03-kube-api-access-dg48j\" (UniqueName: \"kubernetes.io/projected/3481f93a-3c14-43b5-a79c-33e60eaafb03-kube-api-access-dg48j\") pod \"3481f93a-3c14-43b5-a79c-33e60eaafb03\" (UID: \"3481f93a-3c14-43b5-a79c-33e60eaafb03\") "
Oct 30 23:19:22.615583 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:22.615511    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1908b3ff-74fa-4341-93f3-1db56d7a9271-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1908b3ff-74fa-4341-93f3-1db56d7a9271-plugins-config-volume\") pod \"1908b3ff-74fa-4341-93f3-1db56d7a9271\" (UID: \"1908b3ff-74fa-4341-93f3-1db56d7a9271\") "
Oct 30 23:19:22.615861 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:22.615837    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3481f93a-3c14-43b5-a79c-33e60eaafb03-plugins-config-volume" pod "3481f93a-3c14-43b5-a79c-33e60eaafb03" (UID: "3481f93a-3c14-43b5-a79c-33e60eaafb03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:19:22.615965 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:22.615942    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1908b3ff-74fa-4341-93f3-1db56d7a9271-plugins-config-volume" pod "1908b3ff-74fa-4341-93f3-1db56d7a9271" (UID: "1908b3ff-74fa-4341-93f3-1db56d7a9271"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:19:22.618194 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:22.618166    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1908b3ff-74fa-4341-93f3-1db56d7a9271-kube-api-access-gtpfw" pod "1908b3ff-74fa-4341-93f3-1db56d7a9271" (UID: "1908b3ff-74fa-4341-93f3-1db56d7a9271"). InnerVolumeSpecName "kube-api-access-gtpfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:19:22.618446 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:22.618422    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3481f93a-3c14-43b5-a79c-33e60eaafb03-kube-api-access-dg48j" pod "3481f93a-3c14-43b5-a79c-33e60eaafb03" (UID: "3481f93a-3c14-43b5-a79c-33e60eaafb03"). InnerVolumeSpecName "kube-api-access-dg48j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:19:22.716347 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:22.716305    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtpfw\" (UniqueName: \"kubernetes.io/projected/1908b3ff-74fa-4341-93f3-1db56d7a9271-kube-api-access-gtpfw\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:19:22.716347 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:22.716339    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3481f93a-3c14-43b5-a79c-33e60eaafb03-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:19:22.716347 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:22.716351    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg48j\" (UniqueName: \"kubernetes.io/projected/3481f93a-3c14-43b5-a79c-33e60eaafb03-kube-api-access-dg48j\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:19:22.716612 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:22.716362    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1908b3ff-74fa-4341-93f3-1db56d7a9271-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:19:23.111655 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:23.111553    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"]
Oct 30 23:19:23.111845 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:23.111745    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb" podUID="223b2134-2c18-4e58-aa61-c747755ea925" containerName="istio-proxy" containerID="cri-o://9f56737463aa74ce760282ba4ab1574f30be524653490a5d4d433255e6d2271b" gracePeriod=30
Oct 30 23:19:23.205737 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:23.205698    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:23.505498 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:23.505471    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7"
Oct 30 23:19:23.505498 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:23.505493    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7" event={"ID":"1908b3ff-74fa-4341-93f3-1db56d7a9271","Type":"ContainerDied","Data":"942e13e391a93010d260f71b8187eb24c1d6790ca4842832f7897de78fedf0ce"}
Oct 30 23:19:23.506065 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:23.505552    4889 scope.go:122] "RemoveContainer" containerID="ffd607eb635643e0cbb4f8d75dc39499eb0c05bdf0fb205ce8b0bf5c32107ec6"
Oct 30 23:19:23.507039 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:23.507017    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx" event={"ID":"3481f93a-3c14-43b5-a79c-33e60eaafb03","Type":"ContainerDied","Data":"efde2a7cc3371a01a0794bb93c9494e5afa3f17c5022060352939c15c977bca9"}
Oct 30 23:19:23.507107 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:23.507031    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx"
Oct 30 23:19:23.514894 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:23.514859    4889 scope.go:122] "RemoveContainer" containerID="d907650ff56224d18bbac19eaa8399fc849ede2a62af935fe3fd45e2d79496f7"
Oct 30 23:19:23.559922 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:23.559894    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7"]
Oct 30 23:19:23.564612 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:23.564587    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1595eccb-589744bfdc-m69n7"]
Oct 30 23:19:23.601347 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:23.601313    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx"]
Oct 30 23:19:23.605672 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:23.605652    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1595eccb-65879f7ddf-dxpjx"]
Oct 30 23:19:24.501064 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.501032    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:24.773110 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.773035    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1908b3ff-74fa-4341-93f3-1db56d7a9271" path="/var/lib/kubelet/pods/1908b3ff-74fa-4341-93f3-1db56d7a9271/volumes"
Oct 30 23:19:24.773508 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.773479    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3481f93a-3c14-43b5-a79c-33e60eaafb03" path="/var/lib/kubelet/pods/3481f93a-3c14-43b5-a79c-33e60eaafb03/volumes"
Oct 30 23:19:24.914300 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.914263    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"]
Oct 30 23:19:24.914889 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.914869    4889 cpu_manager.go:414] "containerMap: removing container" podUID="3481f93a-3c14-43b5-a79c-33e60eaafb03" containerName="payload-processing"
Oct 30 23:19:24.914941 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.914893    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3481f93a-3c14-43b5-a79c-33e60eaafb03" containerName="payload-processing"
Oct 30 23:19:24.914941 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.914922    4889 cpu_manager.go:414] "containerMap: removing container" podUID="d6df8663-a978-4719-9322-51007ab6b5af" containerName="maas-api"
Oct 30 23:19:24.914941 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.914930    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6df8663-a978-4719-9322-51007ab6b5af" containerName="maas-api"
Oct 30 23:19:24.915051 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.914955    4889 cpu_manager.go:414] "containerMap: removing container" podUID="ca14b6fa-378e-42e3-be2e-3b15de4a82cd" containerName="maas-api"
Oct 30 23:19:24.915051 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.914963    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca14b6fa-378e-42e3-be2e-3b15de4a82cd" containerName="maas-api"
Oct 30 23:19:24.915051 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.914994    4889 cpu_manager.go:414] "containerMap: removing container" podUID="1908b3ff-74fa-4341-93f3-1db56d7a9271" containerName="payload-pre-processing"
Oct 30 23:19:24.915051 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.915002    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1908b3ff-74fa-4341-93f3-1db56d7a9271" containerName="payload-pre-processing"
Oct 30 23:19:24.915202 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.915094    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6df8663-a978-4719-9322-51007ab6b5af" containerName="maas-api"
Oct 30 23:19:24.915202 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.915112    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca14b6fa-378e-42e3-be2e-3b15de4a82cd" containerName="maas-api"
Oct 30 23:19:24.915202 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.915125    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="1908b3ff-74fa-4341-93f3-1db56d7a9271" containerName="payload-pre-processing"
Oct 30 23:19:24.915202 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.915135    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="3481f93a-3c14-43b5-a79c-33e60eaafb03" containerName="payload-processing"
Oct 30 23:19:24.918845 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.918820    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:24.938502 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.938474    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-credential-socket\") pod \"e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") " pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:24.938652 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.938509    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/986c3415-82d0-483c-8089-d2cba2290ab8-istiod-ca-cert\") pod \"e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") " pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:24.938652 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.938530    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-workload-certs\") pod \"e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") " pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:24.938652 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.938551    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-istio-envoy\") pod \"e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") " pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:24.938776 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.938700    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/986c3415-82d0-483c-8089-d2cba2290ab8-istio-podinfo\") pod \"e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") " pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:24.938776 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.938720    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/986c3415-82d0-483c-8089-d2cba2290ab8-istio-token\") pod \"e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") " pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:24.938776 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.938740    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-istio-data\") pod \"e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") " pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:24.938888 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.938820    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sph7z\" (UniqueName: \"kubernetes.io/projected/986c3415-82d0-483c-8089-d2cba2290ab8-kube-api-access-sph7z\") pod \"e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") " pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:24.938923 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.938903    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-workload-socket\") pod \"e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") " pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:24.943384 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.943360    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-9cdd18-openshift-default-dockercfg-dpxxm\""
Oct 30 23:19:24.963615 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:24.959482    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"]
Oct 30 23:19:25.025530 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:25.025433    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:25.039867 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:25.039839    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-istio-data\") pod \"e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") " pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:25.040034 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:25.039876    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sph7z\" (UniqueName: \"kubernetes.io/projected/986c3415-82d0-483c-8089-d2cba2290ab8-kube-api-access-sph7z\") pod \"e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") " pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:25.040034 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:25.039917    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-workload-socket\") pod \"e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") " pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:25.040034 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:25.039940    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-credential-socket\") pod \"e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") " pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:25.040034 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:25.039963    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/986c3415-82d0-483c-8089-d2cba2290ab8-istiod-ca-cert\") pod \"e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") " pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:25.040290 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:25.040099    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-workload-certs\") pod \"e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") " pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:25.040290 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:25.040155    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-istio-envoy\") pod \"e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") " pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:25.040290 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:25.040270    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-istio-data\") pod \"e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") " pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:25.040290 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:25.040283    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/986c3415-82d0-483c-8089-d2cba2290ab8-istio-podinfo\") pod \"e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") " pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:25.040517 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:25.040322    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/986c3415-82d0-483c-8089-d2cba2290ab8-istio-token\") pod \"e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") " pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:25.040517 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:25.040329    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-workload-socket\") pod \"e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") " pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:25.040517 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:25.040447    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-credential-socket\") pod \"e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") " pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:25.040723 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:25.040523    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-workload-certs\") pod \"e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") " pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:25.040830 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:25.040810    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/986c3415-82d0-483c-8089-d2cba2290ab8-istiod-ca-cert\") pod \"e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") " pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:25.042758 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:25.042739    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-istio-envoy\") pod \"e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") " pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:25.042967 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:25.042948    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/986c3415-82d0-483c-8089-d2cba2290ab8-istio-podinfo\") pod \"e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") " pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:25.062313 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:25.062286    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sph7z\" (UniqueName: \"kubernetes.io/projected/986c3415-82d0-483c-8089-d2cba2290ab8-kube-api-access-sph7z\") pod \"e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") " pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:25.063836 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:25.063810    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/986c3415-82d0-483c-8089-d2cba2290ab8-istio-token\") pod \"e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") " pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:25.221533 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:25.221491    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:25.233271 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:25.233248    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:25.551712 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:25.551689    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"]
Oct 30 23:19:25.553091 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:19:25.553058    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod986c3415_82d0_483c_8089_d2cba2290ab8.slice/crio-2612460cea42f3338a0205b2fda077013871b548cc4f4f6d3cf24426f48c9a2f WatchSource:0}: Error finding container 2612460cea42f3338a0205b2fda077013871b548cc4f4f6d3cf24426f48c9a2f: Status 404 returned error can't find the container with id 2612460cea42f3338a0205b2fda077013871b548cc4f4f6d3cf24426f48c9a2f
Oct 30 23:19:25.555293 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:25.555261    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:19:25.555403 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:25.555322    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:19:25.555403 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:25.555351    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:19:26.526008 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:26.525967    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c" event={"ID":"986c3415-82d0-483c-8089-d2cba2290ab8","Type":"ContainerStarted","Data":"14078c8fadde5e5e7f7b807241802b6cfe210aee843086976fc5bb5b8e98fde1"}
Oct 30 23:19:26.526547 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:26.526023    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c" event={"ID":"986c3415-82d0-483c-8089-d2cba2290ab8","Type":"ContainerStarted","Data":"2612460cea42f3338a0205b2fda077013871b548cc4f4f6d3cf24426f48c9a2f"}
Oct 30 23:19:26.597892 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:26.597845    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c" podStartSLOduration=2.597833236 podStartE2EDuration="2.597833236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:26.58430062 +0000 UTC m=+4351.940666936" watchObservedRunningTime="2026-10-30 23:19:26.597833236 +0000 UTC m=+4351.954199542"
Oct 30 23:19:27.233483 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:27.233447    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:27.239388 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:27.239358    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:27.530284 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:27.530255    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:27.531117 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:27.531098    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:28.368676 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.368632    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:19:28.474278 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.474172    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-istio-data\") pod \"223b2134-2c18-4e58-aa61-c747755ea925\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") "
Oct 30 23:19:28.474278 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.474235    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-credential-socket\") pod \"223b2134-2c18-4e58-aa61-c747755ea925\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") "
Oct 30 23:19:28.474278 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.474260    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-workload-certs\") pod \"223b2134-2c18-4e58-aa61-c747755ea925\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") "
Oct 30 23:19:28.474553 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.474330    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/223b2134-2c18-4e58-aa61-c747755ea925-istio-token\" (UniqueName: \"kubernetes.io/projected/223b2134-2c18-4e58-aa61-c747755ea925-istio-token\") pod \"223b2134-2c18-4e58-aa61-c747755ea925\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") "
Oct 30 23:19:28.474553 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.474357    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-istio-envoy\") pod \"223b2134-2c18-4e58-aa61-c747755ea925\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") "
Oct 30 23:19:28.474553 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.474398    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/223b2134-2c18-4e58-aa61-c747755ea925-kube-api-access-qrpl2\" (UniqueName: \"kubernetes.io/projected/223b2134-2c18-4e58-aa61-c747755ea925-kube-api-access-qrpl2\") pod \"223b2134-2c18-4e58-aa61-c747755ea925\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") "
Oct 30 23:19:28.474553 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.474441    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-workload-socket\") pod \"223b2134-2c18-4e58-aa61-c747755ea925\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") "
Oct 30 23:19:28.474553 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.474477    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-istio-data" pod "223b2134-2c18-4e58-aa61-c747755ea925" (UID: "223b2134-2c18-4e58-aa61-c747755ea925"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:19:28.474825 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.474556    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-credential-socket" pod "223b2134-2c18-4e58-aa61-c747755ea925" (UID: "223b2134-2c18-4e58-aa61-c747755ea925"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:19:28.474825 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.474567    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/223b2134-2c18-4e58-aa61-c747755ea925-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/223b2134-2c18-4e58-aa61-c747755ea925-istio-podinfo\") pod \"223b2134-2c18-4e58-aa61-c747755ea925\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") "
Oct 30 23:19:28.474825 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.474604    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/223b2134-2c18-4e58-aa61-c747755ea925-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/223b2134-2c18-4e58-aa61-c747755ea925-istiod-ca-cert\") pod \"223b2134-2c18-4e58-aa61-c747755ea925\" (UID: \"223b2134-2c18-4e58-aa61-c747755ea925\") "
Oct 30 23:19:28.474825 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.474641    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-workload-certs" pod "223b2134-2c18-4e58-aa61-c747755ea925" (UID: "223b2134-2c18-4e58-aa61-c747755ea925"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:19:28.474825 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.474756    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-workload-socket" pod "223b2134-2c18-4e58-aa61-c747755ea925" (UID: "223b2134-2c18-4e58-aa61-c747755ea925"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:19:28.475119 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.474977    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/223b2134-2c18-4e58-aa61-c747755ea925-istiod-ca-cert" pod "223b2134-2c18-4e58-aa61-c747755ea925" (UID: "223b2134-2c18-4e58-aa61-c747755ea925"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:19:28.475119 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.475079    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:19:28.475119 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.475098    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:19:28.475119 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.475115    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:19:28.475542 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.475130    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/223b2134-2c18-4e58-aa61-c747755ea925-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:19:28.475542 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.475143    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:19:28.476864 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.476838    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-istio-envoy" pod "223b2134-2c18-4e58-aa61-c747755ea925" (UID: "223b2134-2c18-4e58-aa61-c747755ea925"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:19:28.477017 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.477004    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/223b2134-2c18-4e58-aa61-c747755ea925-kube-api-access-qrpl2" pod "223b2134-2c18-4e58-aa61-c747755ea925" (UID: "223b2134-2c18-4e58-aa61-c747755ea925"). InnerVolumeSpecName "kube-api-access-qrpl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:19:28.477239 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.477207    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/223b2134-2c18-4e58-aa61-c747755ea925-istio-podinfo" pod "223b2134-2c18-4e58-aa61-c747755ea925" (UID: "223b2134-2c18-4e58-aa61-c747755ea925"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:19:28.477312 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.477253    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/223b2134-2c18-4e58-aa61-c747755ea925-istio-token" pod "223b2134-2c18-4e58-aa61-c747755ea925" (UID: "223b2134-2c18-4e58-aa61-c747755ea925"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:19:28.536502 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.536468    4889 generic.go:356] "Generic (PLEG): container finished" podID="223b2134-2c18-4e58-aa61-c747755ea925" containerID="9f56737463aa74ce760282ba4ab1574f30be524653490a5d4d433255e6d2271b" exitCode=0
Oct 30 23:19:28.536970 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.536518    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"
Oct 30 23:19:28.536970 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.536565    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb" event={"ID":"223b2134-2c18-4e58-aa61-c747755ea925","Type":"ContainerDied","Data":"9f56737463aa74ce760282ba4ab1574f30be524653490a5d4d433255e6d2271b"}
Oct 30 23:19:28.536970 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.536629    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb" event={"ID":"223b2134-2c18-4e58-aa61-c747755ea925","Type":"ContainerDied","Data":"38e8830c4eb64dd5e879bfd0ef42e098cadb720097c8edcd418bc1f507c177bc"}
Oct 30 23:19:28.536970 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.536653    4889 scope.go:122] "RemoveContainer" containerID="9f56737463aa74ce760282ba4ab1574f30be524653490a5d4d433255e6d2271b"
Oct 30 23:19:28.547568 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.547554    4889 scope.go:122] "RemoveContainer" containerID="724d5f3faad685f6c9bc8236a6ff04e6ff69caf01aa30ab23c051afef24f6587"
Oct 30 23:19:28.557174 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.557157    4889 scope.go:122] "RemoveContainer" containerID="9f56737463aa74ce760282ba4ab1574f30be524653490a5d4d433255e6d2271b"
Oct 30 23:19:28.557493 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:19:28.557472    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f56737463aa74ce760282ba4ab1574f30be524653490a5d4d433255e6d2271b\": container with ID starting with 9f56737463aa74ce760282ba4ab1574f30be524653490a5d4d433255e6d2271b not found: ID does not exist" containerID="9f56737463aa74ce760282ba4ab1574f30be524653490a5d4d433255e6d2271b"
Oct 30 23:19:28.557573 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.557502    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f56737463aa74ce760282ba4ab1574f30be524653490a5d4d433255e6d2271b"} err="failed to get container status \"9f56737463aa74ce760282ba4ab1574f30be524653490a5d4d433255e6d2271b\": rpc error: code = NotFound desc = could not find container \"9f56737463aa74ce760282ba4ab1574f30be524653490a5d4d433255e6d2271b\": container with ID starting with 9f56737463aa74ce760282ba4ab1574f30be524653490a5d4d433255e6d2271b not found: ID does not exist"
Oct 30 23:19:28.557573 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.557522    4889 scope.go:122] "RemoveContainer" containerID="724d5f3faad685f6c9bc8236a6ff04e6ff69caf01aa30ab23c051afef24f6587"
Oct 30 23:19:28.557798 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:19:28.557775    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"724d5f3faad685f6c9bc8236a6ff04e6ff69caf01aa30ab23c051afef24f6587\": container with ID starting with 724d5f3faad685f6c9bc8236a6ff04e6ff69caf01aa30ab23c051afef24f6587 not found: ID does not exist" containerID="724d5f3faad685f6c9bc8236a6ff04e6ff69caf01aa30ab23c051afef24f6587"
Oct 30 23:19:28.557909 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.557801    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"724d5f3faad685f6c9bc8236a6ff04e6ff69caf01aa30ab23c051afef24f6587"} err="failed to get container status \"724d5f3faad685f6c9bc8236a6ff04e6ff69caf01aa30ab23c051afef24f6587\": rpc error: code = NotFound desc = could not find container \"724d5f3faad685f6c9bc8236a6ff04e6ff69caf01aa30ab23c051afef24f6587\": container with ID starting with 724d5f3faad685f6c9bc8236a6ff04e6ff69caf01aa30ab23c051afef24f6587 not found: ID does not exist"
Oct 30 23:19:28.576408 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.576379    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/223b2134-2c18-4e58-aa61-c747755ea925-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:19:28.576408 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.576408    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/223b2134-2c18-4e58-aa61-c747755ea925-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:19:28.576612 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.576422    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrpl2\" (UniqueName: \"kubernetes.io/projected/223b2134-2c18-4e58-aa61-c747755ea925-kube-api-access-qrpl2\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:19:28.576612 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.576431    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/223b2134-2c18-4e58-aa61-c747755ea925-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:19:28.592311 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.592287    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"]
Oct 30 23:19:28.600794 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.600772    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1595eccb-openshift-default-84f7ff667-58jtb"]
Oct 30 23:19:28.777492 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:28.776195    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="223b2134-2c18-4e58-aa61-c747755ea925" path="/var/lib/kubelet/pods/223b2134-2c18-4e58-aa61-c747755ea925/volumes"
Oct 30 23:19:29.039268 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:29.039164    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:29.207534 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:29.207485    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:30.347490 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.347460    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd"]
Oct 30 23:19:30.347912 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.347845    4889 cpu_manager.go:414] "containerMap: removing container" podUID="223b2134-2c18-4e58-aa61-c747755ea925" containerName="istio-proxy"
Oct 30 23:19:30.347912 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.347855    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="223b2134-2c18-4e58-aa61-c747755ea925" containerName="istio-proxy"
Oct 30 23:19:30.347912 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.347878    4889 cpu_manager.go:414] "containerMap: removing container" podUID="223b2134-2c18-4e58-aa61-c747755ea925" containerName="istio-proxy"
Oct 30 23:19:30.347912 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.347883    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="223b2134-2c18-4e58-aa61-c747755ea925" containerName="istio-proxy"
Oct 30 23:19:30.348115 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.347951    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="223b2134-2c18-4e58-aa61-c747755ea925" containerName="istio-proxy"
Oct 30 23:19:30.351073 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.351051    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd"
Oct 30 23:19:30.371962 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.371936    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd"]
Oct 30 23:19:30.383182 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.383157    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-9cdd18-serving-cert\""
Oct 30 23:19:30.395775 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.395744    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ada0ba58-1317-4db5-be6b-54af867c6083-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd\" (UID: \"ada0ba58-1317-4db5-be6b-54af867c6083\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd"
Oct 30 23:19:30.395919 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.395898    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzjf6\" (UniqueName: \"kubernetes.io/projected/ada0ba58-1317-4db5-be6b-54af867c6083-kube-api-access-jzjf6\") pod \"maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd\" (UID: \"ada0ba58-1317-4db5-be6b-54af867c6083\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd"
Oct 30 23:19:30.395960 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.395950    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada0ba58-1317-4db5-be6b-54af867c6083-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd\" (UID: \"ada0ba58-1317-4db5-be6b-54af867c6083\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd"
Oct 30 23:19:30.396003 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.395983    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ada0ba58-1317-4db5-be6b-54af867c6083-metrics-certs\") pod \"maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd\" (UID: \"ada0ba58-1317-4db5-be6b-54af867c6083\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd"
Oct 30 23:19:30.496617 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.496583    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ada0ba58-1317-4db5-be6b-54af867c6083-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd\" (UID: \"ada0ba58-1317-4db5-be6b-54af867c6083\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd"
Oct 30 23:19:30.496836 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.496701    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzjf6\" (UniqueName: \"kubernetes.io/projected/ada0ba58-1317-4db5-be6b-54af867c6083-kube-api-access-jzjf6\") pod \"maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd\" (UID: \"ada0ba58-1317-4db5-be6b-54af867c6083\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd"
Oct 30 23:19:30.496836 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.496743    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada0ba58-1317-4db5-be6b-54af867c6083-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd\" (UID: \"ada0ba58-1317-4db5-be6b-54af867c6083\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd"
Oct 30 23:19:30.496836 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:19:30.496755    4889 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-serving-cert: secret "maas-api-e2e-authcfg-http-9cdd18-serving-cert" not found
Oct 30 23:19:30.496836 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.496770    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ada0ba58-1317-4db5-be6b-54af867c6083-metrics-certs\") pod \"maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd\" (UID: \"ada0ba58-1317-4db5-be6b-54af867c6083\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd"
Oct 30 23:19:30.496836 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:19:30.496832    4889 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ada0ba58-1317-4db5-be6b-54af867c6083-maas-api-tls podName:ada0ba58-1317-4db5-be6b-54af867c6083 nodeName:}" failed. No retries permitted until 2026-10-30 23:19:30.996816164 +0000 UTC m=+4356.353182484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ada0ba58-1317-4db5-be6b-54af867c6083-maas-api-tls") pod "maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd" (UID: "ada0ba58-1317-4db5-be6b-54af867c6083") : secret "maas-api-e2e-authcfg-http-9cdd18-serving-cert" not found
Oct 30 23:19:30.497462 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.497434    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada0ba58-1317-4db5-be6b-54af867c6083-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd\" (UID: \"ada0ba58-1317-4db5-be6b-54af867c6083\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd"
Oct 30 23:19:30.498689 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.498667    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:30.499736 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.499717    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ada0ba58-1317-4db5-be6b-54af867c6083-metrics-certs\") pod \"maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd\" (UID: \"ada0ba58-1317-4db5-be6b-54af867c6083\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd"
Oct 30 23:19:30.517920 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.517898    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzjf6\" (UniqueName: \"kubernetes.io/projected/ada0ba58-1317-4db5-be6b-54af867c6083-kube-api-access-jzjf6\") pod \"maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd\" (UID: \"ada0ba58-1317-4db5-be6b-54af867c6083\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd"
Oct 30 23:19:30.677542 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.677446    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:30.778424 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.778391    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l"]
Oct 30 23:19:30.778876 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.778862    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="223b2134-2c18-4e58-aa61-c747755ea925" containerName="istio-proxy"
Oct 30 23:19:30.781288 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.781269    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l"
Oct 30 23:19:30.798893 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.798868    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l"]
Oct 30 23:19:30.799969 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.799940    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-9cdd18\""
Oct 30 23:19:30.844912 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.844882    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44"]
Oct 30 23:19:30.847851 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.847832    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44"
Oct 30 23:19:30.865531 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.865487    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:30.867975 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.867951    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44"]
Oct 30 23:19:30.901182 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.901150    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14058a72-0a3d-411d-82f8-2a845cce1250-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l\" (UID: \"14058a72-0a3d-411d-82f8-2a845cce1250\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l"
Oct 30 23:19:30.901460 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.901423    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swbq4\" (UniqueName: \"kubernetes.io/projected/ae43a34d-46e5-4811-aa45-6c351775d1e5-kube-api-access-swbq4\") pod \"payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44\" (UID: \"ae43a34d-46e5-4811-aa45-6c351775d1e5\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44"
Oct 30 23:19:30.901582 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.901458    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmkm4\" (UniqueName: \"kubernetes.io/projected/14058a72-0a3d-411d-82f8-2a845cce1250-kube-api-access-pmkm4\") pod \"payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l\" (UID: \"14058a72-0a3d-411d-82f8-2a845cce1250\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l"
Oct 30 23:19:30.901671 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.901654    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae43a34d-46e5-4811-aa45-6c351775d1e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44\" (UID: \"ae43a34d-46e5-4811-aa45-6c351775d1e5\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44"
Oct 30 23:19:30.913829 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:30.913776    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-9cdd18-dockercfg-89klj\""
Oct 30 23:19:31.002823 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:31.002795    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swbq4\" (UniqueName: \"kubernetes.io/projected/ae43a34d-46e5-4811-aa45-6c351775d1e5-kube-api-access-swbq4\") pod \"payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44\" (UID: \"ae43a34d-46e5-4811-aa45-6c351775d1e5\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44"
Oct 30 23:19:31.003002 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:31.002831    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmkm4\" (UniqueName: \"kubernetes.io/projected/14058a72-0a3d-411d-82f8-2a845cce1250-kube-api-access-pmkm4\") pod \"payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l\" (UID: \"14058a72-0a3d-411d-82f8-2a845cce1250\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l"
Oct 30 23:19:31.003002 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:31.002863    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ada0ba58-1317-4db5-be6b-54af867c6083-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd\" (UID: \"ada0ba58-1317-4db5-be6b-54af867c6083\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd"
Oct 30 23:19:31.003002 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:31.002891    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae43a34d-46e5-4811-aa45-6c351775d1e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44\" (UID: \"ae43a34d-46e5-4811-aa45-6c351775d1e5\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44"
Oct 30 23:19:31.003002 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:31.002921    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14058a72-0a3d-411d-82f8-2a845cce1250-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l\" (UID: \"14058a72-0a3d-411d-82f8-2a845cce1250\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l"
Oct 30 23:19:31.003702 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:31.003681    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14058a72-0a3d-411d-82f8-2a845cce1250-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l\" (UID: \"14058a72-0a3d-411d-82f8-2a845cce1250\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l"
Oct 30 23:19:31.003817 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:31.003794    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae43a34d-46e5-4811-aa45-6c351775d1e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44\" (UID: \"ae43a34d-46e5-4811-aa45-6c351775d1e5\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44"
Oct 30 23:19:31.005680 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:31.005660    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ada0ba58-1317-4db5-be6b-54af867c6083-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd\" (UID: \"ada0ba58-1317-4db5-be6b-54af867c6083\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd"
Oct 30 23:19:31.023949 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:31.023923    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmkm4\" (UniqueName: \"kubernetes.io/projected/14058a72-0a3d-411d-82f8-2a845cce1250-kube-api-access-pmkm4\") pod \"payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l\" (UID: \"14058a72-0a3d-411d-82f8-2a845cce1250\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l"
Oct 30 23:19:31.024035 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:31.024000    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swbq4\" (UniqueName: \"kubernetes.io/projected/ae43a34d-46e5-4811-aa45-6c351775d1e5-kube-api-access-swbq4\") pod \"payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44\" (UID: \"ae43a34d-46e5-4811-aa45-6c351775d1e5\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44"
Oct 30 23:19:31.093739 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:31.093711    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l"
Oct 30 23:19:31.161100 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:31.161071    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44"
Oct 30 23:19:31.263050 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:31.262975    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd"
Oct 30 23:19:31.320975 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:31.320937    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:31.390852 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:31.390822    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l"]
Oct 30 23:19:31.391324 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:19:31.390950    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14058a72_0a3d_411d_82f8_2a845cce1250.slice/crio-5c0c7eb461ca788ac7092b33ae9fb3e1fc6c04e101ef394c4aacf3eac53e7de2 WatchSource:0}: Error finding container 5c0c7eb461ca788ac7092b33ae9fb3e1fc6c04e101ef394c4aacf3eac53e7de2: Status 404 returned error can't find the container with id 5c0c7eb461ca788ac7092b33ae9fb3e1fc6c04e101ef394c4aacf3eac53e7de2
Oct 30 23:19:31.504087 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:31.504058    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44"]
Oct 30 23:19:31.505017 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:19:31.504992    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae43a34d_46e5_4811_aa45_6c351775d1e5.slice/crio-ef4af6b4c80d5678e5194bb839abf29836d508315b5a6300c95675c46c860cb1 WatchSource:0}: Error finding container ef4af6b4c80d5678e5194bb839abf29836d508315b5a6300c95675c46c860cb1: Status 404 returned error can't find the container with id ef4af6b4c80d5678e5194bb839abf29836d508315b5a6300c95675c46c860cb1
Oct 30 23:19:31.554052 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:31.554026    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44" event={"ID":"ae43a34d-46e5-4811-aa45-6c351775d1e5","Type":"ContainerStarted","Data":"ef4af6b4c80d5678e5194bb839abf29836d508315b5a6300c95675c46c860cb1"}
Oct 30 23:19:31.555622 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:31.555594    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l" event={"ID":"14058a72-0a3d-411d-82f8-2a845cce1250","Type":"ContainerStarted","Data":"cd7a11b69ce8640ff003b9b764bc69424341f4d1f2430285b1594b8c25188681"}
Oct 30 23:19:31.555767 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:31.555641    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l" event={"ID":"14058a72-0a3d-411d-82f8-2a845cce1250","Type":"ContainerStarted","Data":"5c0c7eb461ca788ac7092b33ae9fb3e1fc6c04e101ef394c4aacf3eac53e7de2"}
Oct 30 23:19:31.555767 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:31.555678    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l"
Oct 30 23:19:31.569744 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:31.569719    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd"]
Oct 30 23:19:31.570663 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:19:31.570638    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada0ba58_1317_4db5_be6b_54af867c6083.slice/crio-7f256c9884056f175431276b4d62a742b72d0215e1e8c472b0f11216ef99653a WatchSource:0}: Error finding container 7f256c9884056f175431276b4d62a742b72d0215e1e8c472b0f11216ef99653a: Status 404 returned error can't find the container with id 7f256c9884056f175431276b4d62a742b72d0215e1e8c472b0f11216ef99653a
Oct 30 23:19:31.601649 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:31.601607    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l" podStartSLOduration=1.6015958000000001 podStartE2EDuration="1.6015958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:31.599944979 +0000 UTC m=+4356.956311294" watchObservedRunningTime="2026-10-30 23:19:31.6015958 +0000 UTC m=+4356.957962114"
Oct 30 23:19:32.305345 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:32.305312    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:32.562386 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:32.562295    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44" event={"ID":"ae43a34d-46e5-4811-aa45-6c351775d1e5","Type":"ContainerStarted","Data":"1894d7d9308eaacc6bf91b290c9e932d4229338fcc6b1764f68c33b7be275735"}
Oct 30 23:19:32.562800 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:32.562418    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44"
Oct 30 23:19:32.563803 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:32.563776    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd" event={"ID":"ada0ba58-1317-4db5-be6b-54af867c6083","Type":"ContainerStarted","Data":"ff94a674348a96e4febf67b7082847f1e5021919e2eac539628f94e4869eff37"}
Oct 30 23:19:32.563957 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:32.563816    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd" event={"ID":"ada0ba58-1317-4db5-be6b-54af867c6083","Type":"ContainerStarted","Data":"7f256c9884056f175431276b4d62a742b72d0215e1e8c472b0f11216ef99653a"}
Oct 30 23:19:32.563957 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:32.563840    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd"
Oct 30 23:19:32.604401 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:32.604358    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44" podStartSLOduration=2.604347699 podStartE2EDuration="2.604347699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:32.603166544 +0000 UTC m=+4357.959532859" watchObservedRunningTime="2026-10-30 23:19:32.604347699 +0000 UTC m=+4357.960714007"
Oct 30 23:19:32.647275 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:32.647213    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd" podStartSLOduration=2.319990989 podStartE2EDuration="2.647202137s" totalImagesPullingTime="327.211148ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:32.644871327 +0000 UTC m=+4358.001237642" watchObservedRunningTime="2026-10-30 23:19:32.647202137 +0000 UTC m=+4358.003568447"
Oct 30 23:19:34.126166 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:34.126126    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:35.944138 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:35.944052    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:37.838433 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:37.836841    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:38.574676 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:38.574645    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd"
Oct 30 23:19:39.845910 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:39.845872    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:40.300133 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:19:40.300090    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:19:41.837054 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:41.837014    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:42.566942 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:42.566909    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l"
Oct 30 23:19:43.275619 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:43.274951    4889 patch_prober.go:29] interesting pod/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": read tcp 10.132.0.2:35092->10.132.1.30:15021: read: connection reset by peer" start-of-body=
Oct 30 23:19:43.275619 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:43.275032    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c" podUID="986c3415-82d0-483c-8089-d2cba2290ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": read tcp 10.132.0.2:35092->10.132.1.30:15021: read: connection reset by peer"
Oct 30 23:19:43.572608 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:43.572530    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44"
Oct 30 23:19:43.627066 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:43.627033    4889 generic.go:356] "Generic (PLEG): container finished" podID="986c3415-82d0-483c-8089-d2cba2290ab8" containerID="14078c8fadde5e5e7f7b807241802b6cfe210aee843086976fc5bb5b8e98fde1" exitCode=0
Oct 30 23:19:43.627262 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:43.627104    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c" event={"ID":"986c3415-82d0-483c-8089-d2cba2290ab8","Type":"ContainerDied","Data":"14078c8fadde5e5e7f7b807241802b6cfe210aee843086976fc5bb5b8e98fde1"}
Oct 30 23:19:43.627521 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:43.627507    4889 scope.go:122] "RemoveContainer" containerID="14078c8fadde5e5e7f7b807241802b6cfe210aee843086976fc5bb5b8e98fde1"
Oct 30 23:19:43.628740 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:43.628710    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:19:43.628855 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:43.628791    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:19:43.628855 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:43.628840    4889 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 23:19:43.666192 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:43.666161    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:43.930072 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:43.929944    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:44.640776 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:44.640739    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c" event={"ID":"986c3415-82d0-483c-8089-d2cba2290ab8","Type":"ContainerStarted","Data":"f326bdaf5178c2482cc6fcc38a01728c5934e27b5436a1f571b54ca282dd389c"}
Oct 30 23:19:45.234409 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:45.234377    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:45.234721 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:45.234695    4889 patch_prober.go:29] interesting pod/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 30 23:19:45.234848 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:45.234748    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c" podUID="986c3415-82d0-483c-8089-d2cba2290ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 30 23:19:45.627845 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:45.627807    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:46.234584 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:46.234550    4889 patch_prober.go:29] interesting pod/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 30 23:19:46.234987 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:46.234602    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c" podUID="986c3415-82d0-483c-8089-d2cba2290ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 30 23:19:47.234775 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:47.234732    4889 patch_prober.go:29] interesting pod/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 30 23:19:47.235428 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:47.235392    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c" podUID="986c3415-82d0-483c-8089-d2cba2290ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 30 23:19:47.646097 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:47.646057    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:48.234766 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:48.234727    4889 patch_prober.go:29] interesting pod/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 30 23:19:48.235035 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:48.234999    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c" podUID="986c3415-82d0-483c-8089-d2cba2290ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 30 23:19:49.234667 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:49.234625    4889 patch_prober.go:29] interesting pod/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 30 23:19:49.234968 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:49.234934    4889 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c" podUID="986c3415-82d0-483c-8089-d2cba2290ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 30 23:19:49.562785 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:49.562750    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:50.238486 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:50.238447    4889 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:50.238869 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:50.238849    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:50.239818 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:50.239794    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:19:51.317280 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:51.317178    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:53.119505 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:53.119467    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:53.594381 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:53.594347    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:54.951358 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:54.951317    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:56.747584 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:56.747544    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:19:58.472642 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:19:58.472579    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:00.351963 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:00.351928    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:02.238925 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:02.238887    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:04.049418 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:04.049383    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:05.842423 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:05.842387    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:07.624061 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:07.624019    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:09.334624 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:09.334585    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:10.301057 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:20:10.301018    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:20:11.129725 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:11.129689    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:12.952374 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:12.952336    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:14.736019 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:14.735975    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:16.456147 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:16.456108    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:18.258398 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:18.258342    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:20.126990 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:20.126945    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:21.973278 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:21.973246    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:23.429184 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:23.429148    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:25.334205 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:25.334171    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:27.151358 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:27.151312    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:28.937290 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:28.937253    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:30.754513 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:30.754471    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:32.534896 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:32.534856    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:34.265680 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:34.265642    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:36.036019 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:36.035977    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:37.940787 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:37.940750    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:39.671586 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:39.671538    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:40.300473 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:20:40.300442    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:20:41.358163 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:41.358130    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:43.136584 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:43.136551    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:45.027328 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:45.027279    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:46.859453 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:46.859401    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:48.640454 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:48.640403    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:50.421738 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:50.421702    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:52.236376 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:52.236324    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:53.729808 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:53.729763    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:54.013574 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:54.013535    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:55.863190 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:55.863152    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:57.629865 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:57.629831    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:20:59.447621 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:20:59.447583    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:21:01.257126 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:01.257096    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:21:03.041600 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:03.041561    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:21:04.961397 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:04.961310    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:21:06.836154 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:06.836117    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:21:08.543086 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:08.543047    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:21:10.299996 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:21:10.299961    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:21:10.339046 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:10.339009    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:21:12.124596 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:12.124559    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:21:13.936186 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:13.936155    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:21:14.415062 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:14.415028    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l"]
Oct 30 23:21:14.451561 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:14.451528    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44"]
Oct 30 23:21:14.624557 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:14.624521    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:21:14.796815 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:14.796780    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:21:19.415435 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:19.415398    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l" podUID="14058a72-0a3d-411d-82f8-2a845cce1250" containerName="payload-processing" containerID="cri-o://cd7a11b69ce8640ff003b9b764bc69424341f4d1f2430285b1594b8c25188681" gracePeriod=25
Oct 30 23:21:19.451929 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:19.451893    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44" podUID="ae43a34d-46e5-4811-aa45-6c351775d1e5" containerName="payload-pre-processing" containerID="cri-o://1894d7d9308eaacc6bf91b290c9e932d4229338fcc6b1764f68c33b7be275735" gracePeriod=25
Oct 30 23:21:19.700124 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:19.700101    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l"
Oct 30 23:21:19.734115 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:19.734091    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44"
Oct 30 23:21:19.802962 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:19.802933    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14058a72-0a3d-411d-82f8-2a845cce1250-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14058a72-0a3d-411d-82f8-2a845cce1250-plugins-config-volume\") pod \"14058a72-0a3d-411d-82f8-2a845cce1250\" (UID: \"14058a72-0a3d-411d-82f8-2a845cce1250\") "
Oct 30 23:21:19.803154 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:19.803029    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14058a72-0a3d-411d-82f8-2a845cce1250-kube-api-access-pmkm4\" (UniqueName: \"kubernetes.io/projected/14058a72-0a3d-411d-82f8-2a845cce1250-kube-api-access-pmkm4\") pod \"14058a72-0a3d-411d-82f8-2a845cce1250\" (UID: \"14058a72-0a3d-411d-82f8-2a845cce1250\") "
Oct 30 23:21:19.803320 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:19.803297    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14058a72-0a3d-411d-82f8-2a845cce1250-plugins-config-volume" pod "14058a72-0a3d-411d-82f8-2a845cce1250" (UID: "14058a72-0a3d-411d-82f8-2a845cce1250"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:21:19.803464 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:19.803447    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14058a72-0a3d-411d-82f8-2a845cce1250-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:21:19.805256 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:19.805235    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14058a72-0a3d-411d-82f8-2a845cce1250-kube-api-access-pmkm4" pod "14058a72-0a3d-411d-82f8-2a845cce1250" (UID: "14058a72-0a3d-411d-82f8-2a845cce1250"). InnerVolumeSpecName "kube-api-access-pmkm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:21:19.904101 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:19.904066    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae43a34d-46e5-4811-aa45-6c351775d1e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae43a34d-46e5-4811-aa45-6c351775d1e5-plugins-config-volume\") pod \"ae43a34d-46e5-4811-aa45-6c351775d1e5\" (UID: \"ae43a34d-46e5-4811-aa45-6c351775d1e5\") "
Oct 30 23:21:19.904317 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:19.904142    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae43a34d-46e5-4811-aa45-6c351775d1e5-kube-api-access-swbq4\" (UniqueName: \"kubernetes.io/projected/ae43a34d-46e5-4811-aa45-6c351775d1e5-kube-api-access-swbq4\") pod \"ae43a34d-46e5-4811-aa45-6c351775d1e5\" (UID: \"ae43a34d-46e5-4811-aa45-6c351775d1e5\") "
Oct 30 23:21:19.904404 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:19.904379    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmkm4\" (UniqueName: \"kubernetes.io/projected/14058a72-0a3d-411d-82f8-2a845cce1250-kube-api-access-pmkm4\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:21:19.904479 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:19.904454    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae43a34d-46e5-4811-aa45-6c351775d1e5-plugins-config-volume" pod "ae43a34d-46e5-4811-aa45-6c351775d1e5" (UID: "ae43a34d-46e5-4811-aa45-6c351775d1e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:21:19.906458 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:19.906434    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae43a34d-46e5-4811-aa45-6c351775d1e5-kube-api-access-swbq4" pod "ae43a34d-46e5-4811-aa45-6c351775d1e5" (UID: "ae43a34d-46e5-4811-aa45-6c351775d1e5"). InnerVolumeSpecName "kube-api-access-swbq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:21:20.005605 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.005577    4889 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae43a34d-46e5-4811-aa45-6c351775d1e5-plugins-config-volume\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:21:20.005605 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.005605    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swbq4\" (UniqueName: \"kubernetes.io/projected/ae43a34d-46e5-4811-aa45-6c351775d1e5-kube-api-access-swbq4\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:21:20.102157 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.102122    4889 generic.go:356] "Generic (PLEG): container finished" podID="ae43a34d-46e5-4811-aa45-6c351775d1e5" containerID="1894d7d9308eaacc6bf91b290c9e932d4229338fcc6b1764f68c33b7be275735" exitCode=0
Oct 30 23:21:20.102393 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.102164    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44"
Oct 30 23:21:20.102393 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.102200    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44" event={"ID":"ae43a34d-46e5-4811-aa45-6c351775d1e5","Type":"ContainerDied","Data":"1894d7d9308eaacc6bf91b290c9e932d4229338fcc6b1764f68c33b7be275735"}
Oct 30 23:21:20.102393 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.102273    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44" event={"ID":"ae43a34d-46e5-4811-aa45-6c351775d1e5","Type":"ContainerDied","Data":"ef4af6b4c80d5678e5194bb839abf29836d508315b5a6300c95675c46c860cb1"}
Oct 30 23:21:20.102393 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.102335    4889 scope.go:122] "RemoveContainer" containerID="1894d7d9308eaacc6bf91b290c9e932d4229338fcc6b1764f68c33b7be275735"
Oct 30 23:21:20.103585 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.103562    4889 generic.go:356] "Generic (PLEG): container finished" podID="14058a72-0a3d-411d-82f8-2a845cce1250" containerID="cd7a11b69ce8640ff003b9b764bc69424341f4d1f2430285b1594b8c25188681" exitCode=0
Oct 30 23:21:20.103711 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.103602    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l"
Oct 30 23:21:20.103711 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.103609    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l" event={"ID":"14058a72-0a3d-411d-82f8-2a845cce1250","Type":"ContainerDied","Data":"cd7a11b69ce8640ff003b9b764bc69424341f4d1f2430285b1594b8c25188681"}
Oct 30 23:21:20.103711 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.103639    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l" event={"ID":"14058a72-0a3d-411d-82f8-2a845cce1250","Type":"ContainerDied","Data":"5c0c7eb461ca788ac7092b33ae9fb3e1fc6c04e101ef394c4aacf3eac53e7de2"}
Oct 30 23:21:20.112963 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.112944    4889 scope.go:122] "RemoveContainer" containerID="1894d7d9308eaacc6bf91b290c9e932d4229338fcc6b1764f68c33b7be275735"
Oct 30 23:21:20.113315 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:21:20.113277    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1894d7d9308eaacc6bf91b290c9e932d4229338fcc6b1764f68c33b7be275735\": container with ID starting with 1894d7d9308eaacc6bf91b290c9e932d4229338fcc6b1764f68c33b7be275735 not found: ID does not exist" containerID="1894d7d9308eaacc6bf91b290c9e932d4229338fcc6b1764f68c33b7be275735"
Oct 30 23:21:20.113374 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.113310    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1894d7d9308eaacc6bf91b290c9e932d4229338fcc6b1764f68c33b7be275735"} err="failed to get container status \"1894d7d9308eaacc6bf91b290c9e932d4229338fcc6b1764f68c33b7be275735\": rpc error: code = NotFound desc = could not find container \"1894d7d9308eaacc6bf91b290c9e932d4229338fcc6b1764f68c33b7be275735\": container with ID starting with 1894d7d9308eaacc6bf91b290c9e932d4229338fcc6b1764f68c33b7be275735 not found: ID does not exist"
Oct 30 23:21:20.113374 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.113332    4889 scope.go:122] "RemoveContainer" containerID="cd7a11b69ce8640ff003b9b764bc69424341f4d1f2430285b1594b8c25188681"
Oct 30 23:21:20.121944 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.121928    4889 scope.go:122] "RemoveContainer" containerID="cd7a11b69ce8640ff003b9b764bc69424341f4d1f2430285b1594b8c25188681"
Oct 30 23:21:20.122186 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:21:20.122171    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd7a11b69ce8640ff003b9b764bc69424341f4d1f2430285b1594b8c25188681\": container with ID starting with cd7a11b69ce8640ff003b9b764bc69424341f4d1f2430285b1594b8c25188681 not found: ID does not exist" containerID="cd7a11b69ce8640ff003b9b764bc69424341f4d1f2430285b1594b8c25188681"
Oct 30 23:21:20.122258 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.122192    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd7a11b69ce8640ff003b9b764bc69424341f4d1f2430285b1594b8c25188681"} err="failed to get container status \"cd7a11b69ce8640ff003b9b764bc69424341f4d1f2430285b1594b8c25188681\": rpc error: code = NotFound desc = could not find container \"cd7a11b69ce8640ff003b9b764bc69424341f4d1f2430285b1594b8c25188681\": container with ID starting with cd7a11b69ce8640ff003b9b764bc69424341f4d1f2430285b1594b8c25188681 not found: ID does not exist"
Oct 30 23:21:20.157967 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.157940    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44"]
Oct 30 23:21:20.171980 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.171957    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-9cdd18-7f54b6fc9f-9sg44"]
Oct 30 23:21:20.202892 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.202866    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l"]
Oct 30 23:21:20.204461 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.204443    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-9cdd18-664b9ddfc-56b6l"]
Oct 30 23:21:20.330790 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.330721    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd"]
Oct 30 23:21:20.330937 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.330912    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd" podUID="ada0ba58-1317-4db5-be6b-54af867c6083" containerName="maas-api" containerID="cri-o://ff94a674348a96e4febf67b7082847f1e5021919e2eac539628f94e4869eff37" gracePeriod=30
Oct 30 23:21:20.454869 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.454835    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:21:20.605562 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.605485    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd"
Oct 30 23:21:20.712789 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.712755    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ada0ba58-1317-4db5-be6b-54af867c6083-kube-api-access-jzjf6\" (UniqueName: \"kubernetes.io/projected/ada0ba58-1317-4db5-be6b-54af867c6083-kube-api-access-jzjf6\") pod \"ada0ba58-1317-4db5-be6b-54af867c6083\" (UID: \"ada0ba58-1317-4db5-be6b-54af867c6083\") "
Oct 30 23:21:20.712986 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.712826    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ada0ba58-1317-4db5-be6b-54af867c6083-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ada0ba58-1317-4db5-be6b-54af867c6083-metrics-certs\") pod \"ada0ba58-1317-4db5-be6b-54af867c6083\" (UID: \"ada0ba58-1317-4db5-be6b-54af867c6083\") "
Oct 30 23:21:20.712986 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.712849    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ada0ba58-1317-4db5-be6b-54af867c6083-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ada0ba58-1317-4db5-be6b-54af867c6083-maas-api-tls\") pod \"ada0ba58-1317-4db5-be6b-54af867c6083\" (UID: \"ada0ba58-1317-4db5-be6b-54af867c6083\") "
Oct 30 23:21:20.712986 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.712886    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ada0ba58-1317-4db5-be6b-54af867c6083-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada0ba58-1317-4db5-be6b-54af867c6083-openshift-service-ca\") pod \"ada0ba58-1317-4db5-be6b-54af867c6083\" (UID: \"ada0ba58-1317-4db5-be6b-54af867c6083\") "
Oct 30 23:21:20.713362 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.713330    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ada0ba58-1317-4db5-be6b-54af867c6083-openshift-service-ca" pod "ada0ba58-1317-4db5-be6b-54af867c6083" (UID: "ada0ba58-1317-4db5-be6b-54af867c6083"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:21:20.715063 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.715031    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada0ba58-1317-4db5-be6b-54af867c6083-kube-api-access-jzjf6" pod "ada0ba58-1317-4db5-be6b-54af867c6083" (UID: "ada0ba58-1317-4db5-be6b-54af867c6083"). InnerVolumeSpecName "kube-api-access-jzjf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:21:20.715172 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.715065    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ada0ba58-1317-4db5-be6b-54af867c6083-metrics-certs" pod "ada0ba58-1317-4db5-be6b-54af867c6083" (UID: "ada0ba58-1317-4db5-be6b-54af867c6083"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:21:20.715172 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.715110    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ada0ba58-1317-4db5-be6b-54af867c6083-maas-api-tls" pod "ada0ba58-1317-4db5-be6b-54af867c6083" (UID: "ada0ba58-1317-4db5-be6b-54af867c6083"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:21:20.776022 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.775971    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14058a72-0a3d-411d-82f8-2a845cce1250" path="/var/lib/kubelet/pods/14058a72-0a3d-411d-82f8-2a845cce1250/volumes"
Oct 30 23:21:20.776674 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.776630    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae43a34d-46e5-4811-aa45-6c351775d1e5" path="/var/lib/kubelet/pods/ae43a34d-46e5-4811-aa45-6c351775d1e5/volumes"
Oct 30 23:21:20.813769 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.813736    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzjf6\" (UniqueName: \"kubernetes.io/projected/ada0ba58-1317-4db5-be6b-54af867c6083-kube-api-access-jzjf6\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:21:20.813769 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.813771    4889 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ada0ba58-1317-4db5-be6b-54af867c6083-metrics-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:21:20.813999 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.813797    4889 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ada0ba58-1317-4db5-be6b-54af867c6083-maas-api-tls\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:21:20.813999 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.813808    4889 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada0ba58-1317-4db5-be6b-54af867c6083-openshift-service-ca\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:21:20.989451 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:20.989358    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:21:21.111660 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:21.111622    4889 generic.go:356] "Generic (PLEG): container finished" podID="ada0ba58-1317-4db5-be6b-54af867c6083" containerID="ff94a674348a96e4febf67b7082847f1e5021919e2eac539628f94e4869eff37" exitCode=0
Oct 30 23:21:21.111844 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:21.111682    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd"
Oct 30 23:21:21.111844 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:21.111705    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd" event={"ID":"ada0ba58-1317-4db5-be6b-54af867c6083","Type":"ContainerDied","Data":"ff94a674348a96e4febf67b7082847f1e5021919e2eac539628f94e4869eff37"}
Oct 30 23:21:21.111844 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:21.111759    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd" event={"ID":"ada0ba58-1317-4db5-be6b-54af867c6083","Type":"ContainerDied","Data":"7f256c9884056f175431276b4d62a742b72d0215e1e8c472b0f11216ef99653a"}
Oct 30 23:21:21.111844 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:21.111783    4889 scope.go:122] "RemoveContainer" containerID="ff94a674348a96e4febf67b7082847f1e5021919e2eac539628f94e4869eff37"
Oct 30 23:21:21.121029 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:21.121011    4889 scope.go:122] "RemoveContainer" containerID="ff94a674348a96e4febf67b7082847f1e5021919e2eac539628f94e4869eff37"
Oct 30 23:21:21.121382 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:21:21.121364    4889 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff94a674348a96e4febf67b7082847f1e5021919e2eac539628f94e4869eff37\": container with ID starting with ff94a674348a96e4febf67b7082847f1e5021919e2eac539628f94e4869eff37 not found: ID does not exist" containerID="ff94a674348a96e4febf67b7082847f1e5021919e2eac539628f94e4869eff37"
Oct 30 23:21:21.121446 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:21.121391    4889 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff94a674348a96e4febf67b7082847f1e5021919e2eac539628f94e4869eff37"} err="failed to get container status \"ff94a674348a96e4febf67b7082847f1e5021919e2eac539628f94e4869eff37\": rpc error: code = NotFound desc = could not find container \"ff94a674348a96e4febf67b7082847f1e5021919e2eac539628f94e4869eff37\": container with ID starting with ff94a674348a96e4febf67b7082847f1e5021919e2eac539628f94e4869eff37 not found: ID does not exist"
Oct 30 23:21:21.160339 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:21.160307    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:21:21.166542 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:21.166518    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd"]
Oct 30 23:21:21.171993 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:21.171968    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9cdd18-6db77cc5c7-8ggkd"]
Oct 30 23:21:21.648789 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:21.648751    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:21:22.772416 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:22.772382    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ada0ba58-1317-4db5-be6b-54af867c6083" path="/var/lib/kubelet/pods/ada0ba58-1317-4db5-be6b-54af867c6083/volumes"
Oct 30 23:21:27.656514 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:27.656481    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:21:27.838596 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:27.838558    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:21:27.923070 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:27.922969    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"]
Oct 30 23:21:27.923281 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:27.923254    4889 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c" podUID="986c3415-82d0-483c-8089-d2cba2290ab8" containerName="istio-proxy" containerID="cri-o://f326bdaf5178c2482cc6fcc38a01728c5934e27b5436a1f571b54ca282dd389c" gracePeriod=30
Oct 30 23:21:28.013528 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:28.013492    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:21:33.215285 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.212691    4889 generic.go:356] "Generic (PLEG): container finished" podID="986c3415-82d0-483c-8089-d2cba2290ab8" containerID="f326bdaf5178c2482cc6fcc38a01728c5934e27b5436a1f571b54ca282dd389c" exitCode=0
Oct 30 23:21:33.215285 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.212808    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c" event={"ID":"986c3415-82d0-483c-8089-d2cba2290ab8","Type":"ContainerDied","Data":"f326bdaf5178c2482cc6fcc38a01728c5934e27b5436a1f571b54ca282dd389c"}
Oct 30 23:21:33.215285 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.212903    4889 scope.go:122] "RemoveContainer" containerID="14078c8fadde5e5e7f7b807241802b6cfe210aee843086976fc5bb5b8e98fde1"
Oct 30 23:21:33.695556 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.695532    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:21:33.886614 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.886527    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-istio-envoy\") pod \"986c3415-82d0-483c-8089-d2cba2290ab8\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") "
Oct 30 23:21:33.886614 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.886570    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-credential-socket\") pod \"986c3415-82d0-483c-8089-d2cba2290ab8\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") "
Oct 30 23:21:33.886614 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.886607    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/986c3415-82d0-483c-8089-d2cba2290ab8-kube-api-access-sph7z\" (UniqueName: \"kubernetes.io/projected/986c3415-82d0-483c-8089-d2cba2290ab8-kube-api-access-sph7z\") pod \"986c3415-82d0-483c-8089-d2cba2290ab8\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") "
Oct 30 23:21:33.886882 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.886626    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-workload-socket\") pod \"986c3415-82d0-483c-8089-d2cba2290ab8\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") "
Oct 30 23:21:33.886882 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.886664    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-istio-data\") pod \"986c3415-82d0-483c-8089-d2cba2290ab8\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") "
Oct 30 23:21:33.886882 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.886705    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/986c3415-82d0-483c-8089-d2cba2290ab8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/986c3415-82d0-483c-8089-d2cba2290ab8-istio-podinfo\") pod \"986c3415-82d0-483c-8089-d2cba2290ab8\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") "
Oct 30 23:21:33.886882 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.886751    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-workload-certs\") pod \"986c3415-82d0-483c-8089-d2cba2290ab8\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") "
Oct 30 23:21:33.886882 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.886830    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/986c3415-82d0-483c-8089-d2cba2290ab8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/986c3415-82d0-483c-8089-d2cba2290ab8-istiod-ca-cert\") pod \"986c3415-82d0-483c-8089-d2cba2290ab8\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") "
Oct 30 23:21:33.887267 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.886889    4889 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/986c3415-82d0-483c-8089-d2cba2290ab8-istio-token\" (UniqueName: \"kubernetes.io/projected/986c3415-82d0-483c-8089-d2cba2290ab8-istio-token\") pod \"986c3415-82d0-483c-8089-d2cba2290ab8\" (UID: \"986c3415-82d0-483c-8089-d2cba2290ab8\") "
Oct 30 23:21:33.887267 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.886930    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-workload-socket" pod "986c3415-82d0-483c-8089-d2cba2290ab8" (UID: "986c3415-82d0-483c-8089-d2cba2290ab8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:21:33.887267 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.886960    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-credential-socket" pod "986c3415-82d0-483c-8089-d2cba2290ab8" (UID: "986c3415-82d0-483c-8089-d2cba2290ab8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:21:33.887267 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.887186    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-workload-certs" pod "986c3415-82d0-483c-8089-d2cba2290ab8" (UID: "986c3415-82d0-483c-8089-d2cba2290ab8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:21:33.887267 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.887254    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/986c3415-82d0-483c-8089-d2cba2290ab8-istiod-ca-cert" pod "986c3415-82d0-483c-8089-d2cba2290ab8" (UID: "986c3415-82d0-483c-8089-d2cba2290ab8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:21:33.887560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.887343    4889 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-workload-certs\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:21:33.887560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.887361    4889 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/986c3415-82d0-483c-8089-d2cba2290ab8-istiod-ca-cert\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:21:33.887560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.887374    4889 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-credential-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:21:33.887560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.887387    4889 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-workload-socket\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:21:33.887560 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.887340    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-istio-data" pod "986c3415-82d0-483c-8089-d2cba2290ab8" (UID: "986c3415-82d0-483c-8089-d2cba2290ab8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:21:33.889277 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.889242    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/986c3415-82d0-483c-8089-d2cba2290ab8-istio-podinfo" pod "986c3415-82d0-483c-8089-d2cba2290ab8" (UID: "986c3415-82d0-483c-8089-d2cba2290ab8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:21:33.889437 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.889417    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-istio-envoy" pod "986c3415-82d0-483c-8089-d2cba2290ab8" (UID: "986c3415-82d0-483c-8089-d2cba2290ab8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:21:33.889498 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.889439    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/986c3415-82d0-483c-8089-d2cba2290ab8-kube-api-access-sph7z" pod "986c3415-82d0-483c-8089-d2cba2290ab8" (UID: "986c3415-82d0-483c-8089-d2cba2290ab8"). InnerVolumeSpecName "kube-api-access-sph7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:21:33.889565 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.889545    4889 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/986c3415-82d0-483c-8089-d2cba2290ab8-istio-token" pod "986c3415-82d0-483c-8089-d2cba2290ab8" (UID: "986c3415-82d0-483c-8089-d2cba2290ab8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:21:33.987904 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.987872    4889 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sph7z\" (UniqueName: \"kubernetes.io/projected/986c3415-82d0-483c-8089-d2cba2290ab8-kube-api-access-sph7z\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:21:33.987904 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.987901    4889 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-istio-data\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:21:33.987904 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.987912    4889 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/986c3415-82d0-483c-8089-d2cba2290ab8-istio-podinfo\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:21:33.988134 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.987921    4889 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/986c3415-82d0-483c-8089-d2cba2290ab8-istio-token\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:21:33.988134 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:33.987929    4889 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/986c3415-82d0-483c-8089-d2cba2290ab8-istio-envoy\") on node \"ip-10-0-142-185.ec2.internal\" DevicePath \"\""
Oct 30 23:21:34.218428 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:34.218348    4889 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"
Oct 30 23:21:34.218428 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:34.218355    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c" event={"ID":"986c3415-82d0-483c-8089-d2cba2290ab8","Type":"ContainerDied","Data":"2612460cea42f3338a0205b2fda077013871b548cc4f4f6d3cf24426f48c9a2f"}
Oct 30 23:21:34.218428 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:34.218410    4889 scope.go:122] "RemoveContainer" containerID="f326bdaf5178c2482cc6fcc38a01728c5934e27b5436a1f571b54ca282dd389c"
Oct 30 23:21:34.288421 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:34.288391    4889 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"]
Oct 30 23:21:34.302624 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:34.302597    4889 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-9cdd18-openshift-default-575d9dcd85-h972c"]
Oct 30 23:21:34.774303 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:34.774272    4889 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="986c3415-82d0-483c-8089-d2cba2290ab8" path="/var/lib/kubelet/pods/986c3415-82d0-483c-8089-d2cba2290ab8/volumes"
Oct 30 23:21:40.392369 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:21:40.392298    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:21:41.788419 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:41.788376    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p2flk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:21:41.789104 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:41.789051    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk" podUID="63fa4b71-d6b5-4cd7-b768-6c87b5593a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:21:41.994358 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:41.994323    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-jp9lg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:21:41.994556 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:41.994381    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg" podUID="17ab4abd-dafe-4275-8740-e71ac3d3fd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:21:48.441433 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:48.441392    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:21:48.624282 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:48.624241    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:21:53.818516 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:53.818474    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:21:55.918536 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:55.918507    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 23:21:55.925675 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:21:55.925649    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 23:22:10.301022 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:22:10.300978    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:22:18.042060 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:18.042016    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:22:18.227474 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:18.227430    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:22:18.506601 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:18.506562    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:22:21.625598 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:21.625561    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:22:21.784941 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:21.784898    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:22:26.797267 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:26.796338    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p2flk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:22:26.797267 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:26.796746    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p2flk" podUID="63fa4b71-d6b5-4cd7-b768-6c87b5593a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:22:27.025252 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:27.012314    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-jp9lg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:22:27.025252 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:27.012387    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jp9lg" podUID="17ab4abd-dafe-4275-8740-e71ac3d3fd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:22:27.305396 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:27.305345    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7qjgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:22:27.305621 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:27.305423    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7qjgv" podUID="500a9955-0466-4536-b274-9c669e5ffffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:22:27.344373 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:27.344338    4889 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-v8xjj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:22:27.344571 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:27.344400    4889 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-v8xjj" podUID="c8d3e5a5-6b17-4204-bd2e-5099505f1bc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:22:36.061607 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:36.061531    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-rsjkj_7d34b350-a707-48de-a9d4-2f073385c366/manager/0.log"
Oct 30 23:22:36.544453 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:36.544421    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8586c445d6-vh5wm_457b7ef2-52f7-4589-a3a5-eb76e5d6ae17/manager/0.log"
Oct 30 23:22:37.904017 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:37.903982    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-567764f58f-qdbbq_e6e1d0c3-54bf-490d-bb4e-9bd7039098cf/maas-api/0.log"
Oct 30 23:22:40.301267 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:22:40.301206    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:22:43.353904 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:43.353871    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f865f994d-9lf4d_91d3138c-c623-4574-a608-05088ee9c88b/kube-auth-proxy/0.log"
Oct 30 23:22:43.963764 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:43.963733    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-7qjgv_500a9955-0466-4536-b274-9c669e5ffffa/istio-proxy/0.log"
Oct 30 23:22:44.953633 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:44.953592    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-jp9lg_17ab4abd-dafe-4275-8740-e71ac3d3fd25/istio-proxy/0.log"
Oct 30 23:22:45.139806 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:45.139765    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-p2flk_63fa4b71-d6b5-4cd7-b768-6c87b5593a7c/istio-proxy/0.log"
Oct 30 23:22:45.537737 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:45.537705    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-v8xjj_c8d3e5a5-6b17-4204-bd2e-5099505f1bc4/istio-proxy/0.log"
Oct 30 23:22:46.072476 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:46.072440    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rp6t6_f32ffa1e-863c-4df9-b4d1-f80463ed099c/payload-processing/0.log"
Oct 30 23:22:46.270266 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:46.270214    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-fb457cdb7-m99g8_605c1b60-1fa5-4257-a588-4b2e0a78bcca/router/0.log"
Oct 30 23:22:47.075924 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:47.075895    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4_ef937b28-8694-4a67-8959-8529b98af921/main/0.log"
Oct 30 23:22:47.281163 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:47.281129    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-pxdsm_22e10730-dd08-4c76-8b7c-7b87e23e571d/main/0.log"
Oct 30 23:22:47.668683 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:47.668633    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-qttc8_eba24b6c-74bc-4a07-93c1-ee051a72aba0/main/0.log"
Oct 30 23:22:48.051438 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:48.051402    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8qsr8_f2a5fba2-8fc9-4f83-8f0a-996f76b01825/main/0.log"
Oct 30 23:22:48.252135 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:48.252105    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97_b491f9ed-944c-48f8-95db-a907d8bfc92c/main/0.log"
Oct 30 23:22:48.424830 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:48.424746    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6_437136a8-5481-471f-9ec3-ae970c671a60/main/0.log"
Oct 30 23:22:53.946144 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:53.946108    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:22:58.565908 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:58.565875    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-rsjkj_7d34b350-a707-48de-a9d4-2f073385c366/manager/0.log"
Oct 30 23:22:58.996368 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:22:58.996333    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8586c445d6-vh5wm_457b7ef2-52f7-4589-a3a5-eb76e5d6ae17/manager/0.log"
Oct 30 23:23:00.126137 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:23:00.126107    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-567764f58f-qdbbq_e6e1d0c3-54bf-490d-bb4e-9bd7039098cf/maas-api/0.log"
Oct 30 23:23:04.866424 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:23:04.866395    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f865f994d-9lf4d_91d3138c-c623-4574-a608-05088ee9c88b/kube-auth-proxy/0.log"
Oct 30 23:23:05.435100 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:23:05.435070    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-7qjgv_500a9955-0466-4536-b274-9c669e5ffffa/istio-proxy/0.log"
Oct 30 23:23:06.395843 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:23:06.395810    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-jp9lg_17ab4abd-dafe-4275-8740-e71ac3d3fd25/istio-proxy/0.log"
Oct 30 23:23:06.594034 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:23:06.593999    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-p2flk_63fa4b71-d6b5-4cd7-b768-6c87b5593a7c/istio-proxy/0.log"
Oct 30 23:23:06.976304 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:23:06.976270    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-v8xjj_c8d3e5a5-6b17-4204-bd2e-5099505f1bc4/istio-proxy/0.log"
Oct 30 23:23:07.532950 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:23:07.532909    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rp6t6_f32ffa1e-863c-4df9-b4d1-f80463ed099c/payload-processing/0.log"
Oct 30 23:23:07.726095 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:23:07.726062    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-fb457cdb7-m99g8_605c1b60-1fa5-4257-a588-4b2e0a78bcca/router/0.log"
Oct 30 23:23:08.285668 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:23:08.285639    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-cklq4_ef937b28-8694-4a67-8959-8529b98af921/main/0.log"
Oct 30 23:23:08.488865 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:23:08.488838    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-pxdsm_22e10730-dd08-4c76-8b7c-7b87e23e571d/main/0.log"
Oct 30 23:23:08.885745 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:23:08.885713    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-qttc8_eba24b6c-74bc-4a07-93c1-ee051a72aba0/main/0.log"
Oct 30 23:23:09.297361 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:23:09.297333    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8qsr8_f2a5fba2-8fc9-4f83-8f0a-996f76b01825/main/0.log"
Oct 30 23:23:09.497265 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:23:09.497214    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-cgb97_b491f9ed-944c-48f8-95db-a907d8bfc92c/main/0.log"
Oct 30 23:23:09.697492 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:23:09.697412    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-dz7s6_437136a8-5481-471f-9ec3-ae970c671a60/main/0.log"
Oct 30 23:23:10.300342 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:23:10.300296    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:23:34.250862 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:23:34.250828    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:23:40.300039 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:23:40.299992    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:23:42.564531 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:23:42.564495    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8586c445d6-vh5wm_457b7ef2-52f7-4589-a3a5-eb76e5d6ae17/manager/0.log"
Oct 30 23:23:42.864839 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:23:42.864743    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8586c445d6-vh5wm_457b7ef2-52f7-4589-a3a5-eb76e5d6ae17/manager/0.log"
Oct 30 23:23:43.573870 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:23:43.573846    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rp6t6_f32ffa1e-863c-4df9-b4d1-f80463ed099c/payload-processing/0.log"
Oct 30 23:23:43.790596 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:23:43.790568    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rp6t6_f32ffa1e-863c-4df9-b4d1-f80463ed099c/payload-processing/0.log"
Oct 30 23:23:53.982308 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:23:53.982274    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:24:10.300250 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:24:10.300145    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:24:24.180517 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:24.180472    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:24:24.487142 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:24.487050    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:24:26.484009 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:26.483979    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8586c445d6-vh5wm_457b7ef2-52f7-4589-a3a5-eb76e5d6ae17/manager/0.log"
Oct 30 23:24:26.808912 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:26.808883    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8586c445d6-vh5wm_457b7ef2-52f7-4589-a3a5-eb76e5d6ae17/manager/0.log"
Oct 30 23:24:27.545205 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:27.545177    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rp6t6_f32ffa1e-863c-4df9-b4d1-f80463ed099c/payload-processing/0.log"
Oct 30 23:24:27.730898 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:27.730871    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rp6t6_f32ffa1e-863c-4df9-b4d1-f80463ed099c/payload-processing/0.log"
Oct 30 23:24:28.089589 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.089557    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qktfv/must-gather-2d2p8"]
Oct 30 23:24:28.090043 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.090024    4889 cpu_manager.go:414] "containerMap: removing container" podUID="ada0ba58-1317-4db5-be6b-54af867c6083" containerName="maas-api"
Oct 30 23:24:28.090152 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.090045    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ada0ba58-1317-4db5-be6b-54af867c6083" containerName="maas-api"
Oct 30 23:24:28.090152 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.090074    4889 cpu_manager.go:414] "containerMap: removing container" podUID="986c3415-82d0-483c-8089-d2cba2290ab8" containerName="istio-proxy"
Oct 30 23:24:28.090152 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.090082    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="986c3415-82d0-483c-8089-d2cba2290ab8" containerName="istio-proxy"
Oct 30 23:24:28.090152 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.090095    4889 cpu_manager.go:414] "containerMap: removing container" podUID="ae43a34d-46e5-4811-aa45-6c351775d1e5" containerName="payload-pre-processing"
Oct 30 23:24:28.090152 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.090103    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae43a34d-46e5-4811-aa45-6c351775d1e5" containerName="payload-pre-processing"
Oct 30 23:24:28.090152 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.090126    4889 cpu_manager.go:414] "containerMap: removing container" podUID="14058a72-0a3d-411d-82f8-2a845cce1250" containerName="payload-processing"
Oct 30 23:24:28.090152 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.090132    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14058a72-0a3d-411d-82f8-2a845cce1250" containerName="payload-processing"
Oct 30 23:24:28.090152 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.090147    4889 cpu_manager.go:414] "containerMap: removing container" podUID="986c3415-82d0-483c-8089-d2cba2290ab8" containerName="istio-proxy"
Oct 30 23:24:28.090608 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.090156    4889 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="986c3415-82d0-483c-8089-d2cba2290ab8" containerName="istio-proxy"
Oct 30 23:24:28.090608 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.090274    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="ada0ba58-1317-4db5-be6b-54af867c6083" containerName="maas-api"
Oct 30 23:24:28.090608 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.090293    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="986c3415-82d0-483c-8089-d2cba2290ab8" containerName="istio-proxy"
Oct 30 23:24:28.090608 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.090304    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae43a34d-46e5-4811-aa45-6c351775d1e5" containerName="payload-pre-processing"
Oct 30 23:24:28.090608 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.090316    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="14058a72-0a3d-411d-82f8-2a845cce1250" containerName="payload-processing"
Oct 30 23:24:28.090608 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.090510    4889 memory_manager.go:365] "RemoveStaleState removing state" podUID="986c3415-82d0-483c-8089-d2cba2290ab8" containerName="istio-proxy"
Oct 30 23:24:28.093656 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.093637    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qktfv/must-gather-2d2p8"
Oct 30 23:24:28.124272 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.124246    4889 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qktfv\"/\"default-dockercfg-qgg56\""
Oct 30 23:24:28.126956 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.126938    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qktfv\"/\"kube-root-ca.crt\""
Oct 30 23:24:28.127011 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.126938    4889 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qktfv\"/\"openshift-service-ca.crt\""
Oct 30 23:24:28.153127 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.153080    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:24:28.157337 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.157315    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qktfv/must-gather-2d2p8"]
Oct 30 23:24:28.240704 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.240650    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87srl\" (UniqueName: \"kubernetes.io/projected/6c8b13c9-9d17-425b-ab31-7be9652e60e6-kube-api-access-87srl\") pod \"must-gather-2d2p8\" (UID: \"6c8b13c9-9d17-425b-ab31-7be9652e60e6\") " pod="openshift-must-gather-qktfv/must-gather-2d2p8"
Oct 30 23:24:28.240861 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.240774    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6c8b13c9-9d17-425b-ab31-7be9652e60e6-must-gather-output\") pod \"must-gather-2d2p8\" (UID: \"6c8b13c9-9d17-425b-ab31-7be9652e60e6\") " pod="openshift-must-gather-qktfv/must-gather-2d2p8"
Oct 30 23:24:28.333637 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.333598    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:24:28.341570 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.341517    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87srl\" (UniqueName: \"kubernetes.io/projected/6c8b13c9-9d17-425b-ab31-7be9652e60e6-kube-api-access-87srl\") pod \"must-gather-2d2p8\" (UID: \"6c8b13c9-9d17-425b-ab31-7be9652e60e6\") " pod="openshift-must-gather-qktfv/must-gather-2d2p8"
Oct 30 23:24:28.341648 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.341567    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6c8b13c9-9d17-425b-ab31-7be9652e60e6-must-gather-output\") pod \"must-gather-2d2p8\" (UID: \"6c8b13c9-9d17-425b-ab31-7be9652e60e6\") " pod="openshift-must-gather-qktfv/must-gather-2d2p8"
Oct 30 23:24:28.341882 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.341867    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6c8b13c9-9d17-425b-ab31-7be9652e60e6-must-gather-output\") pod \"must-gather-2d2p8\" (UID: \"6c8b13c9-9d17-425b-ab31-7be9652e60e6\") " pod="openshift-must-gather-qktfv/must-gather-2d2p8"
Oct 30 23:24:28.363659 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.363635    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87srl\" (UniqueName: \"kubernetes.io/projected/6c8b13c9-9d17-425b-ab31-7be9652e60e6-kube-api-access-87srl\") pod \"must-gather-2d2p8\" (UID: \"6c8b13c9-9d17-425b-ab31-7be9652e60e6\") " pod="openshift-must-gather-qktfv/must-gather-2d2p8"
Oct 30 23:24:28.407788 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.407764    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qktfv/must-gather-2d2p8"
Oct 30 23:24:28.705303 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.705273    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qktfv/must-gather-2d2p8"]
Oct 30 23:24:28.706321 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:24:28.706295    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c8b13c9_9d17_425b_ab31_7be9652e60e6.slice/crio-415ec500b3e9ef5e0200d61c59e8ece1ca71474484015c76adde21eddcd05880 WatchSource:0}: Error finding container 415ec500b3e9ef5e0200d61c59e8ece1ca71474484015c76adde21eddcd05880: Status 404 returned error can't find the container with id 415ec500b3e9ef5e0200d61c59e8ece1ca71474484015c76adde21eddcd05880
Oct 30 23:24:28.708382 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:28.708357    4889 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:24:29.092621 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:29.092561    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qktfv/must-gather-2d2p8" event={"ID":"6c8b13c9-9d17-425b-ab31-7be9652e60e6","Type":"ContainerStarted","Data":"415ec500b3e9ef5e0200d61c59e8ece1ca71474484015c76adde21eddcd05880"}
Oct 30 23:24:30.099131 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:30.099097    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qktfv/must-gather-2d2p8" event={"ID":"6c8b13c9-9d17-425b-ab31-7be9652e60e6","Type":"ContainerStarted","Data":"51cb73299f2fd9a2c58ac7ec2d3cb3d6e8b9a1ddf8cd292bcee1c2022ca9510b"}
Oct 30 23:24:30.099541 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:30.099140    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qktfv/must-gather-2d2p8" event={"ID":"6c8b13c9-9d17-425b-ab31-7be9652e60e6","Type":"ContainerStarted","Data":"ba418b814ff2c3e2204f9e170de8e7002e634a5f13e94caf85b39326fd231439"}
Oct 30 23:24:30.143306 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:30.143256    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-qktfv/must-gather-2d2p8" podStartSLOduration=1.382098144 podStartE2EDuration="2.143243421s" totalImagesPullingTime="761.145277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:24:30.142775502 +0000 UTC m=+4655.499141816" watchObservedRunningTime="2026-10-30 23:24:30.143243421 +0000 UTC m=+4655.499609731"
Oct 30 23:24:32.766329 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:32.766294    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ljnhp_1e444015-5346-4106-93aa-ef43eb9d0be6/global-pull-secret-syncer/0.log"
Oct 30 23:24:33.103889 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:33.103806    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-74sl2_00b74f51-6c71-45f2-97af-6aea20b9eeaf/connection-checker/0.log"
Oct 30 23:24:33.236298 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:33.236267    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-dvcrc_b8ee94be-1472-42e7-bf6e-d13e58671eeb/connection-checker/0.log"
Oct 30 23:24:33.370990 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:33.370914    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-k8lw5_3f403b34-f155-49e6-b327-8ad89d00bbc3/connection-checker/0.log"
Oct 30 23:24:33.573489 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:33.569305    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log"
Oct 30 23:24:34.286159 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:34.285941    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-185.ec2.internal_496cd66cfb3bc1cef71f828d2683a76e/haproxy/0.log"
Oct 30 23:24:35.740246 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:35.737253    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:24:40.318553 ip-10-0-142-185 kubenswrapper[4889]: E1030 23:24:40.318504    4889 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwjsl" containerName="extractor"
Oct 30 23:24:42.074398 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:42.074365    4889 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qktfv/perf-node-gather-daemonset-ss97r"]
Oct 30 23:24:42.078771 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:42.078742    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-ss97r"
Oct 30 23:24:42.095777 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:42.095749    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qktfv/perf-node-gather-daemonset-ss97r"]
Oct 30 23:24:42.240462 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:42.240413    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b-proc\") pod \"perf-node-gather-daemonset-ss97r\" (UID: \"cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-ss97r"
Oct 30 23:24:42.240462 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:42.240463    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b-podres\") pod \"perf-node-gather-daemonset-ss97r\" (UID: \"cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-ss97r"
Oct 30 23:24:42.240705 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:42.240497    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b-sys\") pod \"perf-node-gather-daemonset-ss97r\" (UID: \"cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-ss97r"
Oct 30 23:24:42.240705 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:42.240555    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bbc8\" (UniqueName: \"kubernetes.io/projected/cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b-kube-api-access-9bbc8\") pod \"perf-node-gather-daemonset-ss97r\" (UID: \"cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-ss97r"
Oct 30 23:24:42.240705 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:42.240617    4889 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b-lib-modules\") pod \"perf-node-gather-daemonset-ss97r\" (UID: \"cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-ss97r"
Oct 30 23:24:42.341791 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:42.341673    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b-proc\") pod \"perf-node-gather-daemonset-ss97r\" (UID: \"cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-ss97r"
Oct 30 23:24:42.341791 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:42.341730    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b-podres\") pod \"perf-node-gather-daemonset-ss97r\" (UID: \"cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-ss97r"
Oct 30 23:24:42.341791 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:42.341777    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b-sys\") pod \"perf-node-gather-daemonset-ss97r\" (UID: \"cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-ss97r"
Oct 30 23:24:42.342081 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:42.341820    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b-proc\") pod \"perf-node-gather-daemonset-ss97r\" (UID: \"cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-ss97r"
Oct 30 23:24:42.342081 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:42.341839    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bbc8\" (UniqueName: \"kubernetes.io/projected/cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b-kube-api-access-9bbc8\") pod \"perf-node-gather-daemonset-ss97r\" (UID: \"cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-ss97r"
Oct 30 23:24:42.342081 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:42.341889    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b-podres\") pod \"perf-node-gather-daemonset-ss97r\" (UID: \"cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-ss97r"
Oct 30 23:24:42.342081 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:42.341920    4889 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b-lib-modules\") pod \"perf-node-gather-daemonset-ss97r\" (UID: \"cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-ss97r"
Oct 30 23:24:42.342081 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:42.341954    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b-sys\") pod \"perf-node-gather-daemonset-ss97r\" (UID: \"cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-ss97r"
Oct 30 23:24:42.342081 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:42.341982    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b-lib-modules\") pod \"perf-node-gather-daemonset-ss97r\" (UID: \"cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-ss97r"
Oct 30 23:24:42.363440 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:42.363408    4889 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bbc8\" (UniqueName: \"kubernetes.io/projected/cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b-kube-api-access-9bbc8\") pod \"perf-node-gather-daemonset-ss97r\" (UID: \"cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-ss97r"
Oct 30 23:24:42.401415 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:42.401382    4889 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-ss97r"
Oct 30 23:24:42.707519 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:42.707436    4889 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qktfv/perf-node-gather-daemonset-ss97r"]
Oct 30 23:24:42.710064 ip-10-0-142-185 kubenswrapper[4889]: W1030 23:24:42.710038    4889 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcc5d0c5f_6e18_425b_a5d6_a8a856df9d0b.slice/crio-0f09cfc11d5cea0f3aa83f28becc6a0c3c48a0599f83eba62b6e24fb2bea1818 WatchSource:0}: Error finding container 0f09cfc11d5cea0f3aa83f28becc6a0c3c48a0599f83eba62b6e24fb2bea1818: Status 404 returned error can't find the container with id 0f09cfc11d5cea0f3aa83f28becc6a0c3c48a0599f83eba62b6e24fb2bea1818
Oct 30 23:24:43.213323 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:43.213276    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-ss97r" event={"ID":"cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b","Type":"ContainerStarted","Data":"f862bd47e289f6ec457ba0fd58b58537b21cd205f99b53c39e951ce460d388d8"}
Oct 30 23:24:43.213803 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:43.213344    4889 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-ss97r" event={"ID":"cc5d0c5f-6e18-425b-a5d6-a8a856df9d0b","Type":"ContainerStarted","Data":"0f09cfc11d5cea0f3aa83f28becc6a0c3c48a0599f83eba62b6e24fb2bea1818"}
Oct 30 23:24:43.213803 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:43.213405    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-ss97r"
Oct 30 23:24:43.257329 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:43.256915    4889 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-ss97r" podStartSLOduration=1.256895937 podStartE2EDuration="1.256895937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:24:43.255136086 +0000 UTC m=+4668.611502401" watchObservedRunningTime="2026-10-30 23:24:43.256895937 +0000 UTC m=+4668.613262253"
Oct 30 23:24:43.553631 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:43.553595    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-qc4bw_200052bd-2f0a-46d2-8b9b-6bd250b17b05/cluster-monitoring-operator/0.log"
Oct 30 23:24:44.256461 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:44.256428    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4pfw9_a55933c9-03e1-4fac-9260-1869ca54070e/node-exporter/0.log"
Oct 30 23:24:44.375742 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:44.375170    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4pfw9_a55933c9-03e1-4fac-9260-1869ca54070e/kube-rbac-proxy/0.log"
Oct 30 23:24:44.492440 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:44.492415    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4pfw9_a55933c9-03e1-4fac-9260-1869ca54070e/init-textfile/0.log"
Oct 30 23:24:49.235602 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:49.235566    4889 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-ss97r"
Oct 30 23:24:49.356762 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:49.356729    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-n42rm_ea4b0705-2096-41da-b78b-b62590f236e8/networking-console-plugin/0.log"
Oct 30 23:24:50.439508 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:50.439467    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/3.log"
Oct 30 23:24:50.443782 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:50.443725    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qbz82_ea9c808c-62b4-4dc7-9761-bf6ee3700e34/console-operator/4.log"
Oct 30 23:24:52.795985 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:52.795951    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-jh5sc_88fe4dfd-dcda-444d-871b-24ca3fb10d77/volume-data-source-validator/0.log"
Oct 30 23:24:54.167584 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:54.167546    4889 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2fhd5_9d1eecff-b22f-4864-a121-9fb507e783f2/konnectivity-agent/0.log" limit=1024
Oct 30 23:24:55.536705 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:55.536644    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t2gq6_c31a736c-cd3e-42be-b783-fc25436b5302/dns/0.log"
Oct 30 23:24:55.651850 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:55.651818    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t2gq6_c31a736c-cd3e-42be-b783-fc25436b5302/kube-rbac-proxy/0.log"
Oct 30 23:24:56.147045 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:56.147017    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9dqlz_2ec19290-25c4-4b52-8173-22c14e7c9056/dns-node-resolver/0.log"
Oct 30 23:24:57.593805 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:57.593772    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f865f994d-9lf4d_91d3138c-c623-4574-a608-05088ee9c88b/kube-auth-proxy/0.log"
Oct 30 23:24:57.773265 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:57.773214    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-7qjgv_500a9955-0466-4536-b274-9c669e5ffffa/istio-proxy/0.log"
Oct 30 23:24:57.994961 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:57.994934    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nwtzc_10d1cecc-9e30-4354-8b42-60ddceadff44/node-ca/0.log"
Oct 30 23:24:58.080043 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:58.080014    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-jp9lg_17ab4abd-dafe-4275-8740-e71ac3d3fd25/istio-proxy/0.log"
Oct 30 23:24:58.141252 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:58.141177    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-p2flk_63fa4b71-d6b5-4cd7-b768-6c87b5593a7c/istio-proxy/0.log"
Oct 30 23:24:58.261326 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:58.261252    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-v8xjj_c8d3e5a5-6b17-4204-bd2e-5099505f1bc4/istio-proxy/0.log"
Oct 30 23:24:58.434156 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:58.434106    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rp6t6_f32ffa1e-863c-4df9-b4d1-f80463ed099c/payload-processing/0.log"
Oct 30 23:24:58.502137 ip-10-0-142-185 kubenswrapper[4889]: I1030 23:24:58.502104    4889 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-fb457cdb7-m99g8_605c1b60-1fa5-4257-a588-4b2e0a78bcca/router/0.log"